Conference PaperPDF Available

Getting access to what goes on in people's heads?: Reflections on the think-aloud technique

Authors:

Abstract

One of the basic usability testing techniques the HCI community draws on, and which stands out as unique, is thinking aloud. We introduce the many names, uses and modifications of the classical think aloud technique, and ask the rhetorical question: What do researchers think they get when they ask people to think aloud? We answer it by discussing the classical work of Ericsson and Simon(1984), in particular their distinction between vocalisation, verbalisation and retrospective reports and the relation to short term memory. Reintroducing the psychological perspective and the focus on higher order cognitive processes, we argue that access to subjective experience is possible in terms of introspection and describe a technique that invites the user to become a participant in the analysis of his or her own cognitive processes. We suggest that use of think aloud has as a prerequisite explicit descriptions of design, test procedure and framework for analysis. We point out, however, that if the aim is to get access to human thinking, HCI research may benefit from experimental research.
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
897
People’s heads, people’s minds?
- theoretical reflections on thinking aloud
Janni Nielsena, Torkil Clemmensenb, Carsten Yssingc
a,b,cDepartment of Informatics
Copenhagen Business School
Howitzvej 60, 2000 Frederiksberg, DK
jn.inf@cbs.dk
ABSTRACT
The thinking aloud technique stands out as unique in the HCI community, and it is tempting to use as it promises access
to people’s heads, to the cognitive processes. Introducing the many approaches and applications of the thinking aloud
technique, we ask the rhetorical question: What do researchers think they get when they ask people to think aloud? We
answer this by introducing the classical work by K.A. Ericsson and H.A. Simon, in particular their differentiation
between three levels of verbalisations: vocalisation, verbalisation and retrospective reports, including their protocol
analysis. We argue that Ericsson and Simon end up with a model of task directed cognitive processes represented by
verbal protocols reduced to data and conveying an image of the human being as an information processing entity.
Introducing Michael Polanyi´s understanding of perception and sense giving, we argue that the act of thinking aloud is
not a verbalization of thoughts but a process of explanations. The necessity to focus on the transformations of
thoughts to talk and on communication of one’s cognitive processes interferes with the cognitive processes of the task.
Hence, we may talk of a double cognitive load. Verbalisation requires a change in focal awareness from making sense
of the task to constructing meaning in terms of the action.
Keywords: Thinking-aloud, verbal protocols, sense giving, awareness, tacit inference, knowledge.
1. INTRODUCTION
In HCI, researchers often position their choice of techniques in relation to thinking aloud1. Frequently, application of
the technique is mentioned though it is not necessarily described in detail, nor discussed2,3,4,5. In HCI practice,
thinking aloud seems to be one of the most popular techniques. It is often referred to as the usability method, and it is
used both in laboratory settings, workshops and field testing3,6,7. In a survey of methods and techniques used by HCI
practitioners in Denmark (main body of respondents) and researchers (about 25% of respondents), thinking aloud
appeared to be the single most frequently applied technique in testing.8 This should not come as a surprise
internationally the technique is taught as part of the HCI curriculum at many universities and is described in many
textbooks. Dix, Finlay, Abowd and Beale11 have given thinking-aloud the credit of simplicity. They advocate a more
relaxed view of the process, pointing out that the usefulness of thinking aloud is “…largely dependent on the
effectiveness of the recording method and subsequent analysis” (ref. 11, p. 427). Molich9 sees it as a technique ready to
use with proper handling, and Hackos and Redish10 understand it as a very straightforward technique. They suggest
asking the user to think aloud while conducting contextual inquiry site visits. “This is the same technique that we use
during usability testing and the purpose is the same …” (ref. 10, p. 137). Especially Jakob Nielsen12 has been tireless in
promoting the technique and arguing for its benefits. However, Preece13 points to the cognitive load and the added
strain on users as well as the interruptive role of the observer during thinking aloud tests. Preece, Rogers and Sharp14
also comment that the user probably would find it difficult to speak when the task became demanding, and the whole
situation would probably feel awkward. Silence is the likely outcome of the situation, they write. Yet the technique is
tempting because it promises instant results and seems to be cost effective. Only few tests subjects are needed, and the
technique can even be used by non-usability specialists12,15. But most important the technique is assumed to give
access to the cognitive processes during users’ engagement with computers.
Much efforts and high value are placed on user testing. However, there seems to be a lack of research literature
reflecting on users’ application of the technique16,1. How do users experience the technique? What do users think of it?
Teaching graduate students in informatics to think aloud and asking them to reflect on their experience have raised a
number of issues. Students complain they think faster than they can speak, their thought processes are too complex to
be verbalised and thinking aloud interferes with their interaction with the interfaces and the task. Besides, thinking
aloud does not come naturally. Although the students took part in a course on HCI tools and techniques and were
1 Jennifer Branch asked adolescents who were participating in her study on information seeking behaviour, which technique they
preferred: think aloud or think after. Although there were differences in preference, Branch discards this as there were ”no
differences in the extent to which participants spoke freely and openly in the study” (ref. 16, p. 384).
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
898
familiar with user testing and despite explicit assurance that it was the system and not them that was being tested, they
reported, once they became test subjects, that they felt they were being observed, evaluated and judged and that this
influenced their performance. If the interaction with the interfaces repeatedly broke down, they would often explain it
by saying that they were stupid, did not really understand the task or more creatively, suggest that they were unable to
read the screen properly because they needed new glasses.
The aim of the paper is to present a theoretical discussion of the thinking aloud technique. The paper is organised as
follows. We start by introducing thinking aloud, indicating that behind the many uses and names, there are often
modifications, redesign and extensions of the classical thinking aloud technique, and most often no rationale for the
change nor reflection on the consequences are discussed. We ask the rhetorical question: what do researchers think they
get when they ask people to think aloud. We answer the question by introducing the readers to the classical reference to
thinking aloud, the publication by Ericsson and Simon on verbal protocols.1,2 We discuss the distinction between three
levels of verbalisations: vocalisation, verbalisation and retrospective reports. We reflect critically on their reduction of
verbal protocols to “pure data” and reintroduce a psychological perspective. This allows us to identify the essence of
Ericsson and Simon’s approach where they end up with a model of task directed cognitive processes represented by
verbal protocol reduced to data. On this basis, we discuss a number of issues relevant to HCI research.
2. THINKING ALOUD
In the literature, thinking aloud has many names: verbal reports, concurrent verbal protocols, retrospective verbal
protocols, after think aloud and verbal protocols. The lack of clarification is also seen when the concept is placed in
brackets without any explanation as to why3. This is even the case in the title of a paper17. In studies reporting on the
use of thinking aloud, it is often stated that the technique used is different, which seems to mean different from the
classical think aloud technique. However, as shown by Boren and Ramey18, the classical thinking aloud technique by
Ericsson and Simon1 is seldom applied, but often redesigned, modified or extended with other techniques. Wright and
Monk15 explain how they applied the think-aloud technique, involving the user directly in the design. Users were told to
think of themselves as “co-evaluators” of the system. They were asked questions such as “what will the system do
if…?” and “Why did you do that?”. The role of the facilitator in the think aloud sessions was to ask hypothetical
questions that would facilitate speculations about the system and questions about the user’s actions. To acquire insight
into the user’s understanding, the facilitator was also instructed to question the user when he/she asked questions. Buur
and Bagger19 modified think aloud and redesigned it into a user dialogue - to enhance “a dialogue between designers
and users on use, context and technology” (ref. 19, p. 63) and to develop “…workshops where think-aloud was changed
into “co-discovery learning” (ref. 19, p. 65). Van Waes20 combined thinking-aloud with on-line recording of mouse
movements and browser actions. This was supplemented with a questionnaire prior to the test and interviews with
participants about their experience after the test sessions. Nielsen and Christiansen21 combined video recordings of
thinking aloud sessions with subsequent interviews where a priori identified video sequences were the object of a
dialogue interview between researcher and user.
In an interesting study by Koenemann, Carroll, Rosson and Singley3, the aim was to follow up on the distinction
between critical incidents and critical threads in usability testing. Interaction with the computer is seen as a process and
not as isolated steps from breakdown to breakdown. The case involved students engaged in learning processes where
the thinking aloud technique was applied. Videotapes and observation notes enabled a minute-by-minute description of
the learners’ activity captured in episode analysis. There is no description of how think aloud was introduced to the
learners and no description of the test leaders’ role and actions during the test. However, the transcripts included in the
paper present glimpses of how the users sometimes speak in prescriptive comments: “I am about to make this” (ref. 3, p.
246), other times “talk” with the interaction: “I guess these are classes … so … I don’t know what a class is but I …”
(ref. 3, p. 249), and at other times reflect by analogy: ”It’s kind of like hardware design when you build a piece of
hardware you go to the store and buy this chip, and this chip, and this resistor...” (ref. 3, p. 247). With its minute-by-
minute protocol, the approach to verbal reports seems to come close to the classical technique by Ericsson and Simon.
However, Koenemann et al. were not just interested in think aloud, but also in retrospective reports. Ericsson and Simon
were only interested in talk aloud and think aloud. We shall get back to this issue when we discuss their verbal reports.
3. GETTING ACCESS TO PEOPLE’S HEADS?
Think aloud has been used to study search strategies and navigation behaviour of “people looking for detailed
information”20. Think aloud is also used to understand some of the mental processes involved in writing programs 4 and
2 The use of verbal protocols is not limited to usability testing. There is a great deal of literature on applying the technique to study
writing, reading strategies and text comprehension as well as decision-making, and the technique has a long tradition in clinical
psychology.
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
899
to uncover users’ performance and reasoning while engaged in a process of learning Smalltalk3. Also, it has been used
to study students writing and reading processes and the cognitive processes involved in solving a problem.
Most studies rest on the understanding that the techniques allow us access to the cognitive processes, to mental
behaviour and give us insight into thinking. By recording the verbal protocol, you will be able to “…detect cognitive
activities that may not be visible at all” (ref. 10, p. 259). Karsenty22 argues that thinking aloud procedures may be used
to identify the cognitive processes responsible for users’ behaviour, although it cannot be used when users are engaged
in a spoken dialogue - in this case talking with a computer system over the phone. The author points out what seems
obvious: The technique puts a cognitive load on the user requiring a cognitive involvement that may interfere or even
compete with the cognitive requirements of the interaction or the tasks. However, Katalin17 sees no problems with a
double cognitive load. He used think aloud to get access to students´ reading comprehension, and understands text
comprehension as a problem solving activity. He suggests using thinking aloud when students are engaged in reading in
a second language: “Because it involves considerable efforts on the reader’s part to make sense of a text written in an
unfamiliar code” (ref. 17, p. 1). He does not see the two different cognitive processes running simultaneously. One
involves reading and meaning construction from a text written in a foreign language, the other involves talking to
verbalise one’s thoughts. Hence, he does not anticipate any cognitive interference during the process.
Yet, Jennifer Branch16 argues that concurrent verbalisation is problematic when the task involves “a high cognitive
load, when the information is difficult to verbalise because of its form…” (ref. 16, p. 379). Branch has compared the
effectiveness of concurrent and retrospective verbal protocols in her study of adolescents searching for information. The
paper is exceptional because Branch gives a detailed description of the test setting, the facilitator’s role, the instructions
to the test-person and the steps in the test, including details about the analysis of data, the steps and the procedure. She
argues that think aloud provides “the most complete and detailed description of the information-seeking processes …the
specific search terms and decision-making steps…allowed a glimpse into the affective nature of the information-
seeking process as well” (ref. 16, p. 382). Branch points out, however, “the reasons behind the decisions that were made
were often explained in the Think After” (ref.16, p. 389), the interview following the think aloud session.
But what is it that we get access to when asking users to think aloud? Is it really an easy and straightforward technique?
Does it really give us access to what goes on in people’s heads? The technique has been questioned as well as the
theoretical underpinnings. Boren and Ramey18 studied how practitioners actually carried out think aloud sessions and
discuss the practice in relation to the classical work by Ericsson and Simon1. They argue that it is necessary to establish
a firm theoretical grounding and a unified practice before the technique can be called a method.
4. VERBAL REPORTS AS DATA
In the classic text from 1984: Protocol Analysis. Verbal reports as data1, Ericsson and Simon discuss the use of
introspective data3. They wanted to reinstate verbal data as a valid resource for understanding human cognitive
processes to make it (a) possible to use verbal data to verify, not only discover, phenomena of interest, provided (b) that
verbal data was interpreted within a theoretical framework. They stated explicitly that they did NOT try to analyse
peripheral thoughts, daydreaming, thoughts containing mental imagery and thoughts related to feelings which were
excluded.
Ericsson and Simon suggest that most performance measures rely on responses that are psychologically
indistinguishable from verbal reports, as some kind of verbal reporting usually is necessary to understand people’s
actions, even in very simple tasks. They argue that a sentence is the verbal realisation of an idea, and verbs in a sentence
can be used to identify different kinds of information and different cognitive processes. They distinguish between on the
one hand (classical) introspection, retrospective reports, and communication to the experimenter, and on the other hand
verbalisation of currently “heeded” thoughts (thoughts reflecting current attention).4 This differentiation is tied to their
understanding of the short-term memory. The assumption is that everything we know has, at some point, gone through
our short-term memory (STM) and we have been conscious of it. We can verbalise what we are learning while in the
process of learning, and we can verbalise what we know if questioned shortly after the process of learning has taken
place. This is because it is still retained in our short-term memory. However, if there is a time span between learning
and the request to recall, we will produce descriptions and explanations - not a report of our immediate thoughts,
because the information from STM is lost.
The authors’ aim is to study task directed cognitive processes and in their model they distinguish between three kinds:
33 Since then, a revised version of the book has been published (1993), but the 1984 edition is the one that is cited widely.
4 Under think aloud conditions, investigations have shown that 96% of the verbalisations are concurrent. Raw data demonstrates that
heeded thoughts produce more pronouns and fragmented utterances, including more verbs in the present tense, while retrospective
reports produce more verbs in past tense and introspection produces the speaker as the grammatical subject (I, my head), epistemic
verbs (remember, feel, know) and lacks information on the current task 1.
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
900
! Vocalisation of thoughts that are already encoded in the verbal form (talk aloud).
! Verbalisation of a sequence of thoughts held in memory in some other form, e.g. visually (think aloud).
! Retrospective verbalisations (retrospective reports on thoughts not held in memory).
Ericsson and Simon primary interest is talk aloud and think aloud, because these verbalisations express the content of
short-term memory. Retrospective reports are only interesting as cross checks (not as anything that can make
contributions) and only if carried out under conditions that may elicit information from STM. According to the authors,
talk aloud calls for cognitive processes operating directly on orally encoded information. The verbalisation begins
immediately and proceeds as a vocalisation of internal speech. Protocols will take the form of a serial delivery of oral
codes, e.g. numbers, letters, etc. The thinking aloud task calls for both orally encoded information and other kinds of
thoughts held in short term memory. Protocol delivery will take the form of sentences that can be understood as
thoughts without or within the context of other thoughts. Retrospective reports produce output similar to thinking aloud,
except it is more coherent and more prone to errors if compared to what the subject actually saw and did during the
session. For example, “what am I thinking about” suggests that the subject only has infrequent access to intermediate
stages in a thought process because of automation of the process or some kind of meta level or strategic thinking instead
of “report of information attended to” (ref. 1, p. 244).
5. PROTOCOL ANALYSIS
The main focus of Ericsson and Simon’s work was really the formal analysis of verbal reports. In their understanding,
the analysis begins when a given theory has been used to identify the relevant part of the universe to be investigated,
and the data has been collected in a raw form (audiotape from the experimental session). The first part of the analysis
involves writing and editing a transcript of the tape, leaving out the information used to segment the verbal stream. The
second and main part of the analysis is the actual encoding of the verbal reports. The encoding is done by using a priori
determined coding categories, and each segment must be treated independently of the surrounding text.
The raw data is the first step, and Ericsson and Simon found that the emergence of new technology, the audiotape
recorder around 1945, “enhanced our ability to treat verbal protocols as hard data” (ref. 1, p. 4) - either in the form of
audiotape or as verbal transcripts of the recorded tape. The possibility of different researchers going back to the same
verbalisations opened up for the development of more explicit theoretical assumptions based on the interpretation of
data because: ”data do not speak for themselves, especially in a system containing a memory that prevents observations
from ever being exactly replicated. They must always be encoded and interpreted in the framework of a theoretical
structure.” (ref. 1, p. 169).
The second step is the segmentation, where the authors seductively promise simplicity: ”verbal protocols are usually
first segmented into individual statements (assertions, propositions). This simple encoding is seldom difficult or
problematic” (ref. 1, p. 172) and may be done on the basis of content, e.g. ideas or by time. However, the segmentation
is actually more complicated and in other parts of the book Ericsson and Simon go deeply into the methodological
problems of segmenting the verbal data: “…protocols must be divided up so that each segment will constitute one
instance of a general process…”, suggesting that the appropriate cues may be:”…pauses, intonation, contours, etc., as
well as syntactic markers for complete phrases and sentences the cues for segmentation in ordinary discourse” (ref. 1,
p. 205).
The final step is the actual encoding. An encoding scheme should be developed a priori and the vocabulary should be
developed from (a) an initial task analysis and (b) “…from a preliminary examination of the protocols” (ref. 1, p. 266).5
However, the authors note that some tasks have a precise language for communicating, while others do not share a
common vocabulary. This is a recurring problem when trying to establish what goes on in people’s heads, e.g. it is
difficult to talk about cooperation in the maritime domain, because they use a command and control language.
6. THE POSSIBILITY OF INTROSPECTION
In retrospect, the wish of Ericsson and Simon to reinstate verbal data in scientific research was highly recommendable.
However, they developed a protocol of analysis where all “noise” was discarded, leading to extremely rigid reductions.
They assumed that only introspections that were verbalisations of currently heeded thoughts would enhance their
investigation into task directed cognitive processes. This assumption together with their methodology for protocol
analysis reduced the human being to a verbalising, task oriented individual, acting in splendid isolation with no context,
no senses and no emotions to hold her/im. It almost reduced the rich human stream of thoughts to keywords. As a result,
5 As a good rule, Ericsson and Simon suggest that the number of terms in the encoding vocabulary necessary to encode 90% of the
protocol will constitute five or ten per cent of the length of the protocol text. As long as it is not important for the theoretical
predictions, they argue that it is reasonable to use synonyms for terms in the task language.
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
901
the psychological being was lost and substituted by an information processing entity. However, present day usability
studies and research on human thoughts are also interested in how thoughts are mediated by knowledge structures or
artefacts that we design and use6. Also in cognitive modelling, it has been argued that we need more interactive and
environmental modelling that considers the different and complex bodily interaction with the task environment23.
The discussion of the value of introspection has a long history, and Ericsson and Simon’s model should be understood
in relation to Nisbett and Wilson’s paper on verbal reports 24. The goal of Ericsson and Simon as of Nisbett and Wilson
was the same: to distinguish illegitimate introspection from other kinds of more legitimate uses of verbal reports. They
agree that “true introspection”, in the sense of classical, direct introspective access to higher order cognitive processes,
is impossible. However, they have different strategies for improving it. Ericsson and Simon focus on getting access to
lower order cognitive processes like perception and memory. Nisbett and Wilson focus on identifying the condition for
giving accurate verbal reports on higher order cognitive processes such as “thinking”, “affective appraisal” and “action
systems”, and describe introspection as based on people’s causal theories and judgements of plausibility. They
concluded from a review study that people’s reports on their cognitive processes are accurate reports, when influential
stimuli are salient and plausible causes of the responses they produce. But the reports are inaccurate when stimuli are
not salient or plausible causes. In other words, people cannot report accurately on their own cognitive processes: “…we
sometimes tell more than we can know…people sometimes make assertions about mental events to which they may
have no access and these assertions may bear little resemblance to the actual events” (ref. 24, p. 247). But the errors that
people make are systematic and regular and due to the application or generation of causal theories about connections in
the situation. They argue that it makes no sense to talk about a risk of modifying the thought under investigation during
verbal reporting because the thought is the application of causal theories. They see people as having or generating
causal theories that may stem from different, mainly social and cultural, sources: (a) explicit rules in a culture, e.g.
traffic regulations, (b) implicit cultural rules, e.g. dating rules, (c) rules developed by an individual on basis of empirical
observation and (d) in situations where none of the other rules apply, e.g. an individual may develop new rules. Hence
Nisbett and Wilson suggest that if we want more accurate verbal reports, we should consider the psychology of the
people we ask.7
In their understanding, people will use such rules to generate explanations when they are asked to explain or describe
what their thoughts were. They will give an explanation of exactly the kind of introspection of higher order cognitive
processes, which the model of Ericsson and Simon excludes from the protocol analysis.
7. PEOPLE’S HEADS - PEOPLE’S MINDS
Though caution has been voiced about the think-aloud technique, it emerges from the literature as the technique for
getting access to people’s heads: user’s inferences, intuitions, mental models, reasons, reading and meaning
construction, decision making, problem solving or thinking. However, Ericsson and Simon’s interest was perception
and short-term memory: lower order cognitive processes that could be studied through participantsverbal reporting
which ran immediate and concurrent with the thoughts. This meant vocalisations (talk aloud), and verbalisation (think
aloud), and it seems that inner thoughts were assumed to be verbal - not non-verbal understandings or sense-making.
However, Nisbett and Wilson’s focus was higher order cognitive processes such as thinking, emotionally based
appraisal, etc. and with this focus, they laid the ground for a psychological understanding of the subject, and for
retrospective verbalisations (the third level in Ericsson and Simon’s model). Drawing on Nisbett and Wilson’s
observation that we sometimes tell more than we can know even about our own mental processes although we have
no access to them we would like to bring into the discussion a theme relevant to HCI research: the psychological
subject. We do so by introducing Michael Polanyi’s description of perception and other psychological processes25. We
are aware that it is not possible to discuss the understandings of Ericsson and Simon directly with those of Polanyi.
Ericsson and Simon focused on a methodological study of task directed cognitive processes, where Polanyi’s search
was framed within an epistemological discussion. But because Polanyi’s concepts are based on the cognitive processes
of scientific discovery, it allows us to reflect critically upon the scientific understanding of Ericsson and Simon. It also
allows us to contemplate on their understanding of talk aloud and think aloud in experimental testing, because Polanyi’s
model is not limited to scientists. He argues explicitly that the only difference between scientists’ thought processes and
those of ordinary people is that scientists use scientific terms whereas non-scientists use terms from everyday life to
approach a problem. But the general structures of the psychological process the act of knowing - are the same.
6 Ericsson and Simon point to this issue when they write about individual differences in ability to verbalize, e.g. gender, and cite a
study “only highly verbal female subjects were able to give rich and fluent verbalisations in a reverie condition containing mostly
visual images” (ref. 1, p. 250).
7 One way to get accurate reports, according to Nisbett and Wilson, is to allow people only to participate in sessions where the
conditions for being accurate are present, that is: “…those for which the influential factors are plausible and are included in [the
subject’s] a priori causal theories” (ref. 24, p. 250)
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
902
8. PERCEPTION AND OTHER PSYCHOLOGICAL PROCESSES
Where Ericsson and Simon understand perception as a lower order cognitive process already encoded in verbal form -
Polanyi understands perception as the essence of tacit inference and tacit knowing, and as such it is beyond language.
Polanyi argues that perception is fundamental in empirical knowledge. He explains it as a psychological process that
involves observation of external facts “without formal argument and even without explicitly stating the results” (Ref.
25, p. 29). As such, one cannot have access to perception. Let us start by pursuing this understanding of perception.
Polanyi introduces examples from everyday life to discuss perception: visual perceptions, bodily or tactile
perceptions, and symbolic perceptions e.g. the process of reading. Stereovision is the first example, and the author
discusses what we see when we look through a stereoscope. You may want to try to use your hands as binoculars
putting one hand before each eye. Close one eye and look, and you will see one image. In the next step, close/open the
other eye, and you will see another image. In the third step, you look with both eyes through your “binoculars” and a
third image is seen with basis in the difference between the two stereo images. Polanyi says that the two images
function as subsidiaries to you seeing the joint image. But your focus is not on the joint image, it is on the meaning of
the image. He argues that this is a typical structure of tacit knowing; a from-to-knowing from the subsidiaries to the
focal target. The focal target is the meaning of the subsidiaries. He calls it an act of sense giving, which is the semantic
aspect of from-to-knowing that also includes a phenomenal and a functional aspect.
To explain the general structure of knowing, Polanyi introduces: the tacit triad. Below we have tried to illustrate the
triad.
Figure 1. The tacit triad (our illustration)
To understand the tacit triad better, we will turn to the second example of perception; the tactile cognition. This will
also help us to understand the phenomenal aspects of from-to-knowing. Try to imagine that you are blindfolded and
have to feel your way in the dark with the help of e.g. a stick. Polanyi argues that we do not focus on the feeling of the
stick in the hand, this is subsidiary. Our attention is on the far end of the stick. We actually feel the point of the stick as
it touches an obstacle in the dark. He calls this the phenomenal aspect. In this case, it is a phenomenal transformation
at which we have arrived by projecting our senses into the world. He describes the process as indwelling or “pouring
one’s body into”. To make “something function subsidiarily is to interiorise it, or else to pour one’s body into it”(ref.
25, p. 33). We should notice that the focal target, which forms the meaning of the subsidiaries, is at a certain distance
from us. Our senses point into external space just as our actions are projected outward. Hence, the objects of our
conscious attention lie predominantly outside ourselves.8
The last example presents symbolic perception; reading a text. When we read a text, focal awareness is not on each
letter, word or sentence. They function as subsidiaries for what we are focusing on: the meaning. Meaning lies outside
ourselves and is reached through a cognitive process that Polanyi describes as tacit integration. The functional aspect of
the subsidiaries is to bear on the focus of our attention, and through the act of a person, the knower “who integrates one
to the other”, the subsidiaries become related to a focus. This integration is an (unconscious) tacit inference where the
knower “sees” coherence where no coherence is. It is a tacit act of a person and may be valid or wrong. It is a
irreversible process as opposed to explicit inference (conscious intentional) which is reversible. Tacit inference is a
process of integration not deduction. We should notice that there are two kinds of awareness. We may be aware of
things without focusing our attention on them this is a from awareness. We may also be aware of things by focusing
8 Polanyi reflects upon how “our own existence, which we experience, and the world that we observe are interwoven here. Bodily
being…becomes a being in the world, while external observations and projects subsidiarily involving one’s own bodily feelings
become, up to a point … an existential choice” (ref. 25, pp.33-34). He infers that applied to science the existential choices modify
the ground of scientific judgment.
Knower
subsidiary awareness
from
to
to
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
903
on them this is focal awareness. However, the two types of awareness are mutually exclusive, and the knower can
dissolve a triad by looking at the subsidiaries in a different way, e.g. making them the object of her focal attention.
Having uncovered the indeterminacies inherent in from-to-knowing, Polanyi argues that we should acknowledge that:
we can “see beyond established facts”
we have the “capacity of feeling …subtle, virtually invisible, signs of reality”
we have the power to ”…know far more than we can tell.” (ref. 25, p30).
9. GIVING SENSE BEFORE VERBALISING
We may now return to Ericsson and Simon’s understanding of perception as directly related to short-term memory and
already encoded in verbal form. From Polanyis’ point of view, perception is about subsidiary awareness a from
knowledge that cannot be captured by language because the tacit inference on which it is based, is irreversible. Nor can
it be differentiated from focal awareness because it is a from-to knowledge. Hence perception becomes a very
complicated cognitive process and it is the heart of coming to know. The essence is the act of sense-giving, and it is the
knower the ordinary person who in the act of giving sense comes to understand her/is world. Thus knowing is not
the same as verbalisation. On the contrary it is much more than what we can verbalize. Remember that talk aloud and
think aloud are cognitive processes operating directly on oral form. They proceed as a vocalisation of internal speech
according to Ericsson and Simon. If we accept Polanyis’ model, we must reject the idea that thoughts are verbal and
directly accessible in oral speech. On the contrary thoughts are to a large extent tacit. The cognitive processes are tacit
inferences, where the subject sees more than there is to see, establishes coherence where no coherence is and knows
more than s/he can tell us. In the act of knowing, subsidiaries bear on focal awareness towards the focal target; meaning.
But meaning lies outside ourselves in the integration of the subsidiaries (and remember this process is unconscious)
with the focal target. The integration is tacit and as a consequence it cannot be spoken nor captured in verbalisation.
Although you may be the knower who integrates in the act of sense giving, you do it on the basis of indwelling this is
the essence of knowing. As this process cannot be reversed, it cannot be systematically uncovered in a reversible
process through deduction. It is a psychological process, and it is beyond language.
10. THE (SCIENTIFIC) ACT OF KNOWING
If we accept this explanation, it may be enlightening to return to Ericsson and Simon and their verbal protocols analysis.
They start out with raw data, which they, in principle, may receive as recorded tapes and analyse without ever having
been present or taken part in the design of the experiment. The first part involves typing the transcript. From here they
move on to segmentation, and each segment must be treated independently and constitute one instance of a general
process. The segments are identified on the basis of an a priori existing theoretical frame, but they may also be
identified by natural pauses in the verbal stream on the recorder, intonations, time, etc. The last step is the final
encoding. These codes are also a priori defined through initial task analysis and/or preliminary analysis of protocols. In
the final analysis, Ericsson and Simon end up with a model of task directed cognitive processes represented by verbal
protocols reduced to data. Polanyi would reject their understanding of scientific work and that science is founded upon
objectivity, “the solid truth”. On the contrary, he argues that the foundation of science is indeterminable knowledge, and
scientific discovery is based on “indefinable powers of thought” (ref. 25, p. 27). He sees explanations - that which can
be verbalised - as a specific type of knowledge and argues that there are other kinds of knowledge. It is - as already
shown - the other kind of knowledge - tacit knowledge he is interested in He understands tacit knowledge as
indefinable, psychological insights based on tacit inferences - not on rational inferences. He points out that the attempts
to exclude these non-strict rules of inference from the scientific process are misleading and reduce the scientific process
to a logic calculation9. The foundation, he writes, is “the indeterminacy of scientific knowledge” and he identifies three
such indeterminacies:
- Indeterminacy in the content of scientific knowledge. A prerequisite for pursuing an interest is that we a priori accept
something as a fact. But this something always embeds a ”widely indeterminate content” (ref. 25, p28) because it is
impossible to make the content of knowledge fully explicit
- Indeterminacy in coherence, which makes us accept a discovery as true. Discovery is based on seeing coherence
where no coherence is, but the coherence that we see can only be vaguely defined.
- Indeterminacy in the data on which our discovery is based, because data can never be fully identified only aspects of
them, although we may take these aspects to represent the absolute data.
9 This is not a rejection of the logical systematic analysis, which is the other heart of science.
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
904
11. KNOWING MORE THAN WE CAN TELL - TELLING MORE THAN WE CAN KNOW
Cognitive actions cannot be reduced to data processing not even if these actions are task directed. There is no 1:1
relationship between the task and the thoughts interacting with the actions of the subject. On the contrary, if we accept
Polanyi’s model, the mental processes are comprehensive, cognitive activities based on tacit processes and integrated
with tacit knowledge. According to Nisbett and Wilson, mental processes are founded on social and cultural
conventions and knowledge, including our own empirical experiences, the human praxis. Talk aloud, vocalisations, do
not express what goes on in people’s heads, because “we know more than we can tell”. In this process, we build on tacit
knowledge, where indefinable sources of indefinable subsidiaries bear on the object of our focus: meaning - the
subjectively meaningful aspects. In a test situation, this may comprise knowledge and assumptions about the task we are
carrying out, the objects on the screen, the navigation button or panel, the facilitator’s encouragement. As test subjects,
we ascribe assumptions to the situation, the room in which the test is taking place, etc. It comprises the tacit knowledge
that we the knowers draw on (unaware). It functions in our subsidiary awareness and in the act of knowing - we
integrate it with our focal awareness and finally it emerges - through the sense-giving act of the knower as
meaningful. This comprehensive mental process constitutes the foundation. What is expressed in vocalisations or
verbalisations is obviously a poor representation of this thinking process.
When our students complain that they think more than they can verbalise, that they think faster than they can vocalise
it may be because they sense these extremely complex mental processes as almost instant mental processes. They
cannot express what they perceive, but they feel the process running ahead and beyond verbalisation. Nisbett and
Wilson might argue that they make inferences about mental events to which they have no access. However, if we accept
that subsidiary awareness is from (unconscious) to - and inseparable from - focal awareness bearing on meaning then
there is no reason to reject the notion that students can be aware of their mental processes as subsidiaries on a non
verbal level. They do not describe the cognitive process but the way in which they experience it and here they draw
on explicit and implicit conventions and knowledge.
It seems that we may begin to understand why the act of thinking aloud is not a vocalization of thoughts in Ericsson and
Simon’s sense but a process of explanation. It means focusing on the transformation of thoughts to talk and
consequently communicating one’s cognitive processes. The process interferes with the cognitive processes involved in
the given task. Verbalisation requires that focal awareness is turned to generate meaning with what I see and do as a test
person engaged in a task - at the same time that I am engaged in the task. Besides, the meaning of my perceptions and
actions has to be transformed to talk, and even if talk comes immediately and seems to run concurrently with my
thoughts my focal awareness has to change from understanding to verbalisation. Verbalisation is not the same as
understanding. On the contrary, understanding is disturbed because subsidiary and focal awareness keeps changing
object and the two are mutually exclusive. If focus is changed to objects that until then worked at the borderlines of
consciousness, subsidiaries would become something quite different pointing towards a different meaning. Hence, we
may talk of a double cognitive load.
Is perception really a lower order cognitive process or is the distinction between higher and lower order processes a
differentiation that has little meaning? This classification gives second thoughts to contemplating our own activity. We
are no different from ordinary wo/man the act of knowing is the same only we have the terms with which to
describe and analyse the process. In doing this our act would be classified as a higher order cognitive process. But the
interesting process, seeing, integrating, sense-giving, which depends upon subsidiaries bearing on focal awareness
towards meaning would be classified as a lower order cognitive processes!
12. CONCLUDING REMARKS
In the HCI research, the notion of thinking aloud, giving access to human thoughts, has been accepted with
modifications. This is seen when researchers combine think aloud with observations and interviews, add pictures to the
audio in the form of video, combine video with computer recordings of the screen and cursor movements and follow up
with qualitative interviews, etc. Maybe the researchers intuitively know that they have to approach the subject from
many perspectives to get access to people’s thinking. Hence, there is no rejection of think aloud - but hesitations about
what we get when we ask subjects to talk while working on a task. Discussing the understanding of Ericsson and Simon
in relation to Polanyi’s understanding has provided a basis for theoretical reflections on what we get access to and what
we may need to try to capture by use of other techniques. As Polanyi points out with reference to Ryle’s ghost in the
machine, we know the workings of another mind by following it not through a process of inference. Therefore, it is
necessary to apply techniques that help us follow the other mind. Indwelling is the concept that Polanyi uses to
describe the process where one attempts to eliminate the borders between oneself and the other and metaphysically
pour one’s body into. The foundation of this ability is an emotional cognitive process that allows us to project our
being into the being of the other.
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
905
ACKNOWLEDGEMENT
The paper has been discussed in sessions in the HCI research group at the Department of Informatics. Thank you to all
of you. Thanks are also due to the Danish Research Council for the research grant that enables the group to work with
the ideas in the DIT (Design of Interactive Interfaces) project: ref. 9900403.
REFERENCES
1. Ericsson, K. A. & Simon, H. A., Protocol Analysis. Verbal reports as data, Cambridge Massachusets, 1984.
2. Rowley, D. E., “Usability testing in the field: bringing the laboratory to the user”, in Human factors in
computing systems : “Celebrating interdependence”. ACM Press, New York, NY, USA, Boston,
Massachusetts, United States, pp. 252 257, 1994
3. Koenemann-Belliveau, J., Carroll, J. M., Rosson, M. B. & Singley, M. K., “Comparative usability evaluation:
critical incidents and critical threads”, in Human factors in computing systems : “Celebrating
interdependence”, ACM Press, New York, NY, USA, Boston, Massachusetts, United States, pp. 245 251,
1994.
4. Bringham, B., John, R. & Lewis, C., Usability Testing of a Graphical Programming System: Things We
Missed in a Programming Walkthrough, in Human factors in computing systems conference proceedings on
Reaching through technology, New Orleans, Louisiana, United States, pp. 7-12, 1991.
5. Yeo, A., “Cultural Effects in Usability Assessment”, in CHI 98, Doctoral Consortium, pp. 74-76, 1998
6. Kensing, F., Prompted Reflections: A Technique for Understanding complex Work, Interactions, January-
February, pp. 7-15, 1998
7. Nielsen, J., Designers' Decision Making: Design of a User Interface for a Picture Search Database”, Esprit
3066, AMODEUS RP7/WP14, 1992
8. Clemmensen, T., “Interest in theory and method in a Danish community of usability specialists” (in Danish).
Working paper, Copenhagen Business School, Department of Informatics, HCI research group, pp. 1-30,
2002.
9. Molich, R., User friendly systems (in Danish), Teknisk forlag, København, 1994
10. Hackos, J. T. & Redish, J. C., User and Task Analysis for interface Design, Wiley, USA, 1998
11. Dix, A., Finlay, J., Abowd, G. & Beale, R., Human-Computer Interaction, Prentice Hall, 1997
12. Nielsen, J., “Estimating the Number of Subjects Needed for a Thinking Aloud Test”, International Journal of
Human-Computer Studies, 41, no. 3, pp. 385-397, 1994
13. Preece, J., Human-Computer Interaction, Addison-Wesley, England, 1994
14. Preece, J., Rogers, Y. & Sharp, H., Interaction Design: Beyond Human-Computer Interaction, John Wiley &
Son, 2002.
15. Wright, P. C. & Monk, A. F., “The use of think-aloud evaluation methods in design”, ACM SIGCHI Bulletin
December, 23, no. 1, 1990
16. Branch, J. L., “Investigating the information-seeking processes of adolescents: The value of using think alouds
and think afters”, Library & Information Science Research, 22, no. 4, pp. 371-392, 2000
17. Katalin, E., “"Please, keep talking": The ‘think-aloud' method in second language reading research”, Novelty,
7, no. 3, 2000.
18. Boren, M. T. & Ramey, J., “Thinking aloud: Reconciling theory and practice”, IEEE Transactions on
Professional Communication, 43, no. 3, pp. 261-278, 2000
19. Buur, J. & Bagger, K., “Replacing usability testing with user dialogue - How a Danish manufacturing company
enhanced its product design process by supporting user participation”, Communications of the ACM, 42, no. 5,
pp. 63-66, 1999.
20. Waes, L. v., “Evaluating on-line and off-line searching behavior using thinking-aloud protocols to detect
navigation barriers”, in Proceedings of the sixteenth annual international conference on Computer
documentation, ACM Press, New York, NY, USA, Quebec, Canada, pp. 180 183, 1998.
21. Nielsen, J. & Christiansen, N. “Mindtape: A Tool for Reflection in Participatory Design”, in Participatory
Design, New York, pp. 303 313, 2000.
22. Karsenty, L., “Adapting verbal protocol methods to investigate speech systems use”, Applied Ergonomics, 32,
no. 1, pp. 15-22, 2001
23. Ritter, F. E. & Young, R. M., “Embodied models as simulated users: introduction to this special issue on using
cognitive models to improve interface design”, International Journal of Human-Computer Studies, 55, no. 1,
pp. 1-14, 2001
24. Nisbett, R. E. & Wilson, T. D. (1977), “Telling More Than We Can Know: Verbal Reports on Mental
Processes”, Psychological Review, vol. 84, no. 3, pp. 231-259.
25. Polanyi, Michael (1967) Logic and Psychology, American Psychologist 23, p. 27-43
Proceedings of the APCHI 2002 (5th Asia Pacific Conference on Computer Human Interaction): User Interaction
technology in the 21st Century, 2002, China, Science Press, Beijing (pp 897-906)
906
... They concluded with a semi-structured interview (13 minutes). Throughout, the researcher observed and recorded participant reactions and invited participants to think-aloud [98] as they used the system. If a participant offered feedback on some portion of the interface during their investigation (e.g., offered detailed feedback on the Time Series View), the researcher did not ask about this topic again during the semi-structured interview. ...
... Then, during the final meeting, each historian investigated some specific query by analyzing the comprehensive set of all mentions using the Document Feed and Document Viewer. During each meeting, the researcher observed each historian and invited the historian to think aloud [98] as they used the system. The researcher also asked the historian to describe their findings and explain how ClioQuery helped or did not help answer their research question. ...
Preprint
Historians and archivists often find and analyze the occurrences of query words in newspaper archives, to help answer fundamental questions about society. But much work in text analytics focuses on helping people investigate other textual units, such as events, clusters, ranked documents, entity relationships, or thematic hierarchies. Informed by a study into the needs of historians and archivists, we thus propose ClioQuery, a text analytics system uniquely organized around the analysis of query words in context. ClioQuery applies text simplification techniques from natural language processing to help historians quickly and comprehensively gather and analyze all occurrences of a query word across an archive. It also pairs these new NLP methods with more traditional features like linked views and in-text highlighting to help engender trust in summarization techniques. We evaluate ClioQuery with two separate user studies, in which historians explain how ClioQuery's novel text simplification features can help facilitate historical research. We also evaluate with a separate quantitative comparison study, which shows that ClioQuery helps crowdworkers find and remember historical information. Such results suggest possible new directions for text analytics in other query-oriented settings.
... This study aimed to explore (a) the acceptability of a workplace canteen intervention combining pre-ordering and prompts with PACE messaging to swap to lower-energy alternatives and (b) the factors influencing the acceptance of lower-energy swaps offered. (Ericsson & Simon, 1993;Nielsen et al., 2002) was employed, which involved participants narrating to me their thoughts and decisions, as they navigated the pre-ordering website that was intended for use in the Bupa Eats pilot trial. Think-aloud approaches have been widely used in the study of consumer decision making and in human-computer userexperience (Bettman & Park, 1980;Perski et al., 2017;Stefano et al., 2010). ...
Thesis
The average adult in the UK consumes 200-300 calories beyond their Guideline Daily Amount. For working adults, more than one-third of calories are consumed in the workplace, making this an important environment for intervention. This thesis makes a contribution to the academic literature, by improving our understanding of how and when offering lower-energy alternatives (‘swaps’) is effective, and to public health by refining an intervention which could be delivered in workplace canteens. Two scoping reviews were conducted (studies 1 & 2) and pointed towards the potential effectiveness of pre-ordering lunch and offering healthier swaps as strategies that may help to improve the healthfulness of food and drink choices. When offering lower energy swaps for snacks and non-alcoholic drinks, studies 3 (n=449) and 4 (n=3,481) recruited samples of UK adults in employment to test the effect of different messages on the acceptance of swaps in an experimental online canteen. The results indicated that messages focusing on the lower-energy content of swaps offered may be an effective and acceptable approach. When highlighting the energy content of swaps offered, increasing the interpretability of this information, by providing physical activity calorie equivalent information (PACE) (i.e., the number of minutes walking required to expend the energy contained) further increased the acceptance of snack and drink swaps offered. In study 5, an online version of a real-world canteen was developed and the intervention (prompts to swaps accompanied by a PACE message) was due to be tested in a real-world trial with the healthcare organisation Bupa. However, due to Covid-19, it was tested qualitatively with employees (n=30) of this organisation across the full lunch menu to provide insights about the factors perceived to influence swap acceptance and the acceptability of the intervention. Swap acceptance was facilitated by the provision of PACE information, and swap similarity in terms of taste, texture, and expected satiety as well as the perception that alternatives provided meaningful energy savings. Overall, the intervention was viewed as an acceptable approach to help reduce energy intake in the workplace. Following refinements to the intervention, Study 6 tested the effect of offering lower-energy swaps with and without PACE messages on the energy of hypothetical lunches pre-ordered with a representative online sample of working adults (n=2,150). Offering swaps with and without a PACE message was found to significantly reduce average energy ordered at lunch compared to when no swaps were offered, the PACE message was more acceptable, and there was no evidence of significant interactions between intervention efficacy and participant characteristics. Offering lower-energy swaps in the workplace when employees pre-order is an acceptable and promising intervention to reduce the energy of foods and drinks ordered. Future work should replicate this research in real-world settings.
... Once the participant could remember and perform all gestures correctly, the experimental protocol began. Throughout the entire test, the participants were encouraged to think-aloud (Nielsen, Clemmensen, and Yssing 2002). The participants followed a recipe, controlled the music while doing so, and rejected an incoming call using HSoS. ...
Article
The kitchen is one of the busiest and messiest hubs of a home, where the hands are usually spoiled with food. In this setting, gestural interaction can offer several advantages: efficient, intuitive, and touch-free orchestration of interactive devices. Yet, research scarcely investigates the user's perspective on gesture-based systems in the kitchen and lacks designs developed through a user-centred process. With the contribution of 234 participants, we conducted formative studies that investigated users' expectations, perceived benefits, and concerns of gesture-based control in the kitchen. These studies guided our conceptual design ‘2HandTouch’, an information display controlled by Hand Specific On-Skin Gestures (HSoS). Then, we evaluated the user experience of this design through a summative user study. We introduced four perceived benefits of HSoS use in the kitchen that has not been visible in such detail. These are i) hygiene, (ii) time management, (iii) imminent control, and (iv) an uninterrupted cooking practice. We also provided broader implications of our work for designing gesture-controlled devices in the kitchen context.
... There is a wide range of techniques used in usability testing, including questionnaires, think-aloud observation, and interview-based techniques [37]. Usability testing for HeLP-Diabetes involved users thinking aloud while undertaking prespecified tasks (eg, finding specific information or using one of the self-monitoring tools), a technique common and unique to HCI [38]. Usability testing helped to optimize the navigation and interactive features of HeLP-Diabetes. ...
Article
BACKGROUND: Digital health research encompasses methods from human-computer interaction and health research. OBJECTIVE: This paper aims to describe how these methods were combined to develop HeLP-Diabetes: Starting Out, a web-based structured education program for people newly diagnosed with type 2 diabetes. METHODS: The development process consisted of three phases: initial design for effectiveness, optimization for usability, and in the wild testing in the National Health Service with people newly diagnosed with type 2 diabetes, and further revisions. We adopted an iterative user-centered approach and followed steps from the human-computer interaction design life cycle and the Medical Research Council guidelines on developing and evaluating complex interventions. RESULTS: The initial design process resulted in an 8-session program containing information and behavior change techniques targeting weight loss, being more active, and taking medication. The usability testing was highlighted at an early stage, where changes needed to be made to the language and layout of the program. The in the wild testing provided data on uptake of and barriers to use. The study suggested low uptake and completion of the program, but those who used it seemed to benefit from it. The qualitative findings suggested that barriers to use included an expectation that the program would take too long. This informed refinements to the program. CONCLUSIONS: The use of interdisciplinary methods resulted in an iterative development process and refinements to the program that were based on user needs and data on uptake. The final intervention was more suitable for a definitive evaluation than the initial version. The description of our approach informs other digital health researchers on how to make interventions more sensitive to user needs.
... First, two coders independently coded the verbalizations to identify different comprehension thought processes. As think-aloud verbalizations may contain fragmented utterances (Charters, 2003;Nielsen, Clemmensen, & Yssing, 2002), we referred to the screen recordings whenever needed to better understand participants' verbalizations during the entire analysis. Afterward, the two coders discussed their codes to gain a consensus on their codes. ...
Preprint
Full-text available
Older adults have been hit disproportionally hard by the COVID-19 pandemic. One critical way for older adults to minimize the negative impact of COVID-19 and future pandemics is to stay informed about its latest information, which has been increasingly presented through online interactive visualizations (e.g., live dashboards and websites). Thus, it is imperative to understand how older adults interact with and comprehend online COVID-19 interactive visualizations and what challenges they might encounter to make such visualizations more accessible to older adults. We adopted a user-centered approach by inviting older adults to interact with COVID-19 interactive visualizations while at the same time verbalizing their thought processes using a think-aloud protocol. By analyzing their think-aloud verbalizations, we identified four types of thought processes representing how older adults comprehended the visualizations and uncovered the challenges they encountered. Furthermore, we also identified the challenges they encountered with seven common types of interaction techniques adopted by the visualizations. Based on the findings, we present design guidelines for making interactive visualizations more accessible to older adults.
Chapter
A conversational system is a core component needed to create a chatbot, one incredibly popular form of conversational interface, without any programming knowledge, made possible by different toolkits and cloud-based Natural Language Understanding (NLU) services. However, these NLU services provide curatorial tools that do not always follow mental models resembling that of the content curators. Moreover, we present an interface for content curators that are aligned with concepts management embedded in taxonomies. We conducted a workshop to understand content creators’ work practices, created a prototype interface, and conducted a user evaluation to identify usability and productivity aspects. Based on our prototype’s evaluation insights, we provide implications for designing user interfaces of NLU services to support curators’ work practices and enhance collaboration.
Article
Historians and archivists often find and analyze the occurrences of query words in newspaper archives, to help answer fundamental questions about society. But much work in text analytics focuses on helping people investigate other textual units, such as events, clusters, ranked documents, entity relationships, or thematic hierarchies. Informed by a study into the needs of historians and archivists, we thus propose ClioQuery , a text analytics system uniquely organized around the analysis of query words in context. ClioQuery applies text simplification techniques from natural language processing to help historians quickly and comprehensively gather and analyze all occurrences of a query word across an archive. It also pairs these new NLP methods with more traditional features like linked views and in-text highlighting to help engender trust in summarization techniques. We evaluate ClioQuery with two separate user studies, in which historians explain how ClioQuery ’s novel text simplification features can help facilitate historical research. We also evaluate with a separate quantitative comparison study, which shows that ClioQuery helps crowdworkers find and remember historical information. Such results suggest possible new directions for text analytics in other query-oriented settings.
Article
Background: The Kids Intracranial Injury Decision Support tool for Traumatic Brain Injury (KIIDS-TBI) tool is a validated risk prediction model for managing children with mild traumatic brain injuries (mTBI) and intracranial injuries. Electronic clinical decision support (CDS) may facilitate the clinical implementation of this evidence-based guidance. Objective: Our objective was to evaluate the acceptability and usability of an electronic CDS tool for managing children with mTBI and intracranial injuries. Methods: Emergency medicine and neurosurgery physicians (10 each) from 10 hospitals in the United States were recruited to participate in usability testing of a novel CDS prototype in a simulated electronic health record environment. Testing included a think-aloud protocol, an acceptability and usability survey, and a semi-structured interview. The prototype was updated twice during testing to reflect user feedback. Usability problems recorded in the videos were categorized using content analysis. Interview transcripts were analyzed using thematic analysis. Results: Among the 20 participants, most worked at teaching hospitals (80%), freestanding children's hospitals (95%), and level-1 trauma centers (75%). During the two prototype updates, problems with clarity of terminology and navigating through the CDS interface were identified and corrected. Corresponding to these changes, the number of usability problems decreased from 35 in phase 1 to 8 in phase 3 and the number of mistakes made decreased from 18 (phase 1) to 2 (phase 3). Through the survey, participants found the tool easy to use (90%), useful for determining a patient's level of care (95%), and likely to improve resource use (90%) and patient safety (79%). Interview themes related to the CDS's ability to support evidence-based decision-making and improve clinical workflow proposed implementation strategies and potential pitfalls. Conclusion: After iterative evaluation and refinement, the KIIDS-TBI CDS tool was found to be highly usable and useful for aiding the management of children with mTBI and intracranial injuries.
Article
Full-text available
This paper reports on the explorative use of video recordings in studies of distributed collaboration. The primary goal is for the analyst to acquire a better understanding of the inter-action. A concurrent aim is to take the Scandinavian System Development tradition one step further and explore, together with users, what goes on in their minds. For this purpose, a method called Mindtape has been developed, the essence of which is to review a priori selected video sequences in a dia-logue with the users. Mindtape enhances the analyst's understanding and helps to clarify or reject interpretations. At the same time, it shows promising possibilities as a tool for mutual learning proc-esses. The users become aware of obstacles, new behaviors and meeting cultures evolving. By allowing users and ana-lysts to get a better understanding of the process of interac-tion, Mindtape may also be used to inform design.
Article
Full-text available
Reviews evidence which suggests that there may be little or no direct introspective access to higher order cognitive processes. Ss are sometimes (a) unaware of the existence of a stimulus that importantly influenced a response, (b) unaware of the existence of the response, and (c) unaware that the stimulus has affected the response. It is proposed that when people attempt to report on their cognitive processes, that is, on the processes mediating the effects of a stimulus on a response, they do not do so on the basis of any true introspection. Instead, their reports are based on a priori, implicit causal theories, or judgments about the extent to which a particular stimulus is a plausible cause of a given response. This suggests that though people may not be able to observe directly their cognitive processes, they will sometimes be able to report accurately about them. Accurate reports will occur when influential stimuli are salient and are plausible causes of the responses they produce, and will not occur when stimuli are not salient or are not plausible causes. (86 ref)
Article
Evidence is reviewed which suggests that there may be little or no direct introspective access to higher order cognitive processes. Subjects are sometimes (a) unaware of the existence of a stimulus that importantly influenced a response, (b) unaware of the existence of the response, and (c) unaware that the stimulus has affected the response. It is proposed that when people attempt to report on their cognitive processes, that is, on the processes mediating the effects of a stimulus on a response, they do not do so on the basis of any true introspection. Instead, their reports are based on a priori, implicit causal theories, or judgments about the extent to which a particular stimulus is a plausible cause of a given response. This suggests that though people may not be able to observe directly their cognitive processes, they will sometimes be able to report accurately about them. Accurate reports will occur when influential stimuli are salient and are plausible causes of the responses they produce, and will not occur when stimuli are not salient or are not plausible causes.
Article
The absence of strict criteria on which to base the acceptance of science shows that the acceptance of scientific knowledge is based on nonstrict criteria. This nonstrict logic rests to an extent on informal elements derived from psychological observations. In these informal elements of science, one can discern the indeterminacy of scientific knowledge, including its indeterminate content and that the coherence that lends validity to a discovery can be only vaguely defined and the data on which a discovery rests are not fully definable. However, forms of tacit knowing, tacit inference and indwelling (interiorization) permit one to see beyond established facts. The lack of clear knowledge of tacit knowing has misguided both philosophy and the knowledge of other minds, universal terms, principles of explanation, and empirical generalization. "Logic" can be seen as the rules for reaching valid conclusions from premises assumed to be true. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Five participants completed retrospective and concurrent verbal protocols (Think Afters and Think Alouds) to evaluate the methods for the study of the information-seeking processes of 12–15-year-old students using Microsoft Encarta 98. After a short training session in the Think Aloud Method, they completed four activities of differing complexity. The data provided support for the use of verbal protocol analysis to uncover information-seeking processes of these students. The amount of data generated during Think Alouds and Think Afters depended on the difficulty of the questions and the number of “dead ends” encountered by the students. The Think Alouds and Think Afters provided data about the behavioral, cognitive, and affective processes. Think Afters provide different data than the Think Alouds, and both are important to understanding how adolescents interact with CD-ROM encyclopedias. Participants were able to find the information to answer each of the four search activity questions but used a variety of search terms, categories, and strategies. Some participants reached a level of frustration after a number of “dead ends” and needed encouragement from the researcher. The frustration was more apparent in the younger participants. Effective reading strategies are very important to successful location and evaluation of information in CD-ROM encyclopedias.
Article
An abstract is not available.
Article
This paper reports on two studies in which teams of two or three trainee designers evaluated a user interface by observing a user working through some set tasks. These users were instructed to think aloud as they worked. The instruction received by the designers took the form of a brief how-to-do-it manual. Study 1 demonstrates that this method is effective. Study 2 found that more problems were detected by the designers of the system than other groups. Also, designers cannot predict the problems users will experience in advance of user testing.