No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
The Use of Think-aloud Methods in Qualitative Research An Introduction to Think-aloud Methods
Abstract
Think-aloud is a research method in which participants speak aloud any words in their mind as they complete a task. A review of the literature has shown that think-aloud research methods have a sound theoretical basis and provide a valid source of data about participant thinking, especially during language based activities. However, a researcher needs to design a process which takes into account a number of concerns, by selecting a suitable task, a role for the researcher, a source of triangulation, and, most importantly, an appropriate method of interpretation. This paper argues that think-aloud research can be effectively interpreted through a qualitative lens. A qualitative approach also has implications for the choice of participant(s) and the treatment of the data. Participants should be treated as quasi-researchers, and their efforts rewarded with reciprocity.
... Intervention usability and user experience and satisfaction will be formatively evaluated via interviews. The first 30 minutes of the 90-minute interview will involve think-aloud [40] review of the user interface of the course, with participants screen-sharing their experience of using the online platform. The think-aloud method is a robust and flexible research technique to test usability by providing valuable and reliable information of users' cognitive processes while completing a task [40]. ...
... The first 30 minutes of the 90-minute interview will involve think-aloud [40] review of the user interface of the course, with participants screen-sharing their experience of using the online platform. The think-aloud method is a robust and flexible research technique to test usability by providing valuable and reliable information of users' cognitive processes while completing a task [40]. This think-aloud task will be followed by a 60-minute qualitative interview, with interview questions developed using the NASSS framework of digital health implementation [33] to prompt discussion of multiple issues within this timeframe (see Appendices 1-5 for interview protocols). ...
... communication partners such as paid support workers, friends, family) will be invited to participate in further implementation interviews. This number of users interviewed is consistent with the think-aloud methods described in the study design [40], which are used to refine the usability of the courses. The internationally recognised industry standard [50] is for a minimum of five users to undergo a formative usability interview evaluation [50], because only so many users are required to identify up to 90% of usability issues [51] before there are diminishing returns for the product cycle [50]. ...
Background
Acquired brain injuries (ABIs) commonly cause cognitive-communication disorders, which can have a pervasive psychosocial impact on a person’s life. More than 135 million people worldwide currently live with ABI, and this large and growing burden is increasingly surpassing global rehabilitation service capacity. A web-based service delivery model may offer a scalable solution. The Social Brain Toolkit is an evidence-based suite of 3 web-based communication training interventions for people with ABI and their communication partners. Successful real-world delivery of web-based interventions such as the Social Brain Toolkit requires investigation of intervention implementation in addition to efficacy and effectiveness.
Objective
The aim of this study is to investigate the implementation and effectiveness of the Social Brain Toolkit as a web-based service delivery model.
Methods
This is a mixed methods, prospective, hybrid type 2 implementation-effectiveness study, theoretically underpinned by the Nonadoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework of digital health implementation. We will document implementation strategies preemptively deployed to support the launch of the Social Brain Toolkit interventions, as well as implementation strategies identified by end users through formative evaluation of the Social Brain Toolkit. We will prospectively observe implementation outcomes, selected on the basis of the NASSS framework, through quantitative web analytics of intervention use, qualitative and quantitative pre- and postintervention survey data from all users within a specified sample frame, and qualitative interviews with a subset of users of each intervention. Qualitative implementation data will be deductively analyzed against the NASSS framework. Quantitative implementation data will be analyzed descriptively. We will obtain effectiveness outcomes through web-based knowledge tests, custom user questionnaires, and formal clinical tools. Quantitative effectiveness outcomes will be analyzed through descriptive statistics and the Reliable Change Index, with repeated analysis of variance (pretraining, posttraining, and follow-up), to determine whether there is any significant improvement within this participant sample.
Results
Data collection commenced on July 2, 2021, and is expected to conclude on June 1, 2022, after a 6-month sample frame of analytics for each Social Brain Toolkit intervention. Data analysis will occur concurrently with data collection until mid-2022, with results expected for publication late 2022 and early 2023.
Conclusions
End-user evaluation of the Social Brain Toolkit’s implementation can guide intervention development and implementation to reach and meet community needs in a feasible, scalable, sustainable, and acceptable manner. End user feedback will be directly incorporated and addressed wherever possible in the next version of the Social Brain Toolkit. Learnings from these findings will benefit the implementation of this and future web-based psychosocial interventions for people with ABI and other populations.
Trial Registration
Australia and New Zealand Clinical Trials Registry ACTRN12621001170819; https://anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12621001170819, Australia and New Zealand Clinical Trials Registry ACTRN12621001177842; https://anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12621001177842, Australia and New Zealand Clinical Trials Registry ACTRN12621001180808; https://anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12621001180808
International Registered Report Identifier (IRRID)
DERR1-10.2196/31995
... Webpages and link sharing are familiar concepts to most people. 4. Allow multiple users to connect to a single display. ...
... 2. An in-game interactions of participant groups were then evaluated during their interaction experiences with Coin Chaser using the Think-Aloud technique. The Think-aloud technique requires participants to narrate their thoughts to the researcher while completing a set task [4]. To evaluate participant groups as if they were interacting with the display in an imagined public space, we navigated to the game and asked participants to interact with the game while also narrating their thoughts throughout the process of initial smartphone connection establishment, gameplay experience, and eventual controller disconnection. ...
We introduce SmartControllerJS, a new JavaScript library for fast, cost-effective designing of web applications controlled via everyday smartphones. At its core, SmartControllerJS establishes a connection between two webpages, one page running on a desktop browser and the other on the user's smartphone. The smartphone webpage loads a controller interface allowing users to control a web application running on their computer's browser. The SmartControllerJS framework enables fast iteration loops when designing interactive user experiments because it has minimal friction and allows for scaling, while having no running costs. We first describe how this library is built, how it can be used, and provide interactive examples. We then present two games designed for public screens along with results from user studies evaluating acceptability and ease of use. Finally, we implement a custom controller based on user feedback and introduce connection monitoring tools. We believe SmartControllerJS can accelerate the design of interactive experiments for researchers in Human-Computer Interaction, and be a useful tool for educational projects. To experience the various demos, we recommend reading this work on a desktop computer with your smartphone in hand. The library and the demos are available at https://github.com/SmartControllerJS
... We are now poised for usability testing, as per the fifth and final step in the ICRR development process (Figure 1). This will be achieved using a 'think-aloud method,' [46][47][48] where CR staff will anonymously enter a patient's data into the ICRR demonstration site on a videoconferencing platform, and then undergo a semi-structured interview. Interview audio-recordings will be transcribed, and analyzed using content analysis concurrently with collection. ...
Introduction: The International Council of Cardiovascular Prevention and Rehabilitation (ICCPR) is developing a registry (ICRR) specifically for low-resource settings, where the burden of cardiovascular diseases is greatest and the need for program development highest. Herein we describe the development process, including the variable selection process.
Method: Following a literature search on registry best practices, a stepwise model for ICRR development was identified. Then, based on recommendations by Core Outcome Set STAndards for Development (COS-STAD), we underwent a process to identify variables. All available CR registries were contacted to request their data dictionaries, reviewed CR quality indicators and guideline recommendations, and searched for common data elements and core outcome sets; 35 unique variables (including patient-reported outcomes) were selected for potential inclusion. Twenty-one purposively-identified stakeholders and experts agreed to serve on a Delphi panel. Panelists rated the variables in an online survey, and suggested potential additional variables; A webcall was held to reach consensus on which to include/exclude. Next, panelists provided input to finalize each variable definition, and rated which associated indicators should be used for benchmarking in registry dashboards and a patient lay summary; a second consensus call was held. A 1-month public comment period
ensued.
Results: First, registry objectives and governance were approved by ICCPR, including data quality and access policies. The protocol was developed, for public posting. For variable selection, the overall mean rating was 6.1 ± 0.3/7; 12 were excluded, some of which were moved to a program survey, and others were revised. Two variables were added in an annual follow-up, resulting in 13 program and 16 patient-reported variables. Legal advice was sought to finalize ICRR agreements. Ethics approvals were obtained. Usability testing is now being initiated.
Conclusion: It is hoped this will serve to harmonize CR assessment internationally and enable quality improvement in CR delivery in low-resource settings.
... In addition to questions about the diagnostic process, the instrument included items that assessed trust in the clinician, feelings about the visit, and health literacy. 27 Initial versions of the instrument were reviewed using the thinkaloud method 28 with patients at a separate site (n ¼ 5) to assess for clarity. To obtain further feedback, the instrument was also presented to Geisinger's Patient and Family Advisory Council, the Patient Experience department leadership, and a multidisciplinary system-wide committee focused on improving clinical diagnosis. ...
Background
The 21st Century Cures Act mandates patients’ access to their electronic health record (EHR) notes. To our knowledge, no previous work has systematically invited patients to proactively report diagnostic concerns while documenting and tracking their diagnostic experiences through EHR-based clinician note review.
Objective
To test if patients can identify concerns about their diagnosis through structured evaluation of their online visit notes.
Methods
In a large integrated health system, patients aged 18–85 years actively using the patient portal and seen between October 2019 and February 2020 were invited to respond to an online questionnaire if an EHR algorithm detected any recent unexpected return visit following an initial primary care consultation (“at-risk” visit). We developed and tested an instrument (Safer Dx Patient Instrument) to help patients identify concerns related to several dimensions of the diagnostic process based on notes review and recall of recent “at-risk” visits. Additional questions assessed patients’ trust in their providers and their general feelings about the visit. The primary outcome was a self-reported diagnostic concern. Multivariate logistic regression tested whether the primary outcome was predicted by instrument variables.
Results
Of 293 566 visits, the algorithm identified 1282 eligible patients, of whom 486 responded. After applying exclusion criteria, 418 patients were included in the analysis. Fifty-one patients (12.2%) identified a diagnostic concern. Patients were more likely to report a concern if they disagreed with statements “the care plan the provider developed for me addressed all my medical concerns” [odds ratio (OR), 2.65; 95% confidence interval [CI], 1.45–4.87) and “I trust the provider that I saw during my visit” (OR, 2.10; 95% CI, 1.19–3.71) and agreed with the statement “I did not have a good feeling about my visit” (OR, 1.48; 95% CI, 1.09–2.01).
Conclusion
Patients can identify diagnostic concerns based on a proactive online structured evaluation of visit notes. This surveillance strategy could potentially improve transparency in the diagnostic process.
... Participants represented a variety of L1s and academic majors. Nine participants were enrolled in the study, a sample size fitting for a think-aloud study (Charters, 2003;Denzin & Lincoln, 2005). Table 1 provides basic information about them. ...
Academic reading in English requires processing of challenging advanced texts for comprehension. Highly proficient second language (L2) readers, such as graduate students, are expected to interpret complex texts fluently and efficiently for academic success, yet the field has little evidence on how these readers process texts. Through two think‐aloud reading sessions and pre‐ and postreading interviews, this study closely examined nine graduate students' interactions with texts that had low and high levels of discourse cohesion. The focus was on the readers' meaning‐construction process in relation to the texts' cohesion. The researchers analyzed these data qualitatively using a grounded theory approach that revealed text‐processing activities of proficient L2 readers grouped at local and global levels. Although all readers experienced these two levels with both texts, some differences were found in their processing between two texts, especially in reading the low‐cohesive text. The findings of the study provide implications for reading instructors and teacher researchers in understanding how advanced L2 readers construct meaning and draw on text discourse as they read texts in English and potentially in conceptualizing what novice L2 readers must do to progress.
... Adopters and implementers were not part of the focus group discussion that was conducted among the dental interns to allow dental interns express their opinions comfortably. The fact that a think-aloud method will always exclude some thought processes that are not held long enough to be expressed in working memory, a follow up interview is commonly recommended to add in-depth information of participants thought processes and to allow interviewees to validate researchers' interpretation of their think-aloud utterances [20]. Individual one-to-one semi structured qualitative interviews were conducted for the adopters and implementers to reflect on their perceptions toward the ISAC intervention (see Additional file 1). ...
Objective
We aimed to explore dentists’ perceptions toward the implementation of a comprehensive intervention (ISAC) for the early detection and prevention of oral cancer in a dental clinic.
Methods
The ISAC intervention was presented to ten purposefully sampled dentists in Jazan Dental School (JDS). Participating dental interns were asked to practice the ISAC intervention whilst thinking aloud. A semi-structured interview technique was used to allow free expression of participants’ perceptions related to the ISAC intervention and to control the flow of topics. Fleuren’s framework theory informed the analysis. The interviews were transcribed verbatim and analyzed using the deductive-inductive framework analysis.
Results
Practicing the ISAC intervention was perceived to enhance the early detection and prevention of oral cancer. Serving community needs and engaging community groups were perceived to be related to a high relevance and compatibility of the ISAC intervention. Being a comprehensive intervention with well-defined objectives and being built on relevant data from the participants’ community and having dentists as a target group were the perceived relative advantages of the ISAC intervention compared to other programs. A supportive environment, gender-concordance, use of regional trainers, standard examination form and collaboration with other sectors were perceived to be the facilitators. Competition with clinical time, use of different examination forms and low organizational leader interests were perceived as impeding factors against effective implementation in a real-world context. Reward, easy to practice, feeling confidence and satisfaction, advertisement as well as use of a role model approach were perceived to be motivating factors.
Conclusions
Integrating data from representers of different participant groups during intervention conceptualization and development are critical for the intervention compatibility and acceptability. The study findings showed the opportunities of intertwining the intrinsic motivators of satisfaction and altruism existing in the target group and the extrinsic motivator of official diagnostic skill, certification that may boost and sustain the behavior change. Intervention features that influence perceived relevance, compatibility, relative advantage and motivation may be of great importance for intervention practice.
... Usability sessions were moderated by the UX researcher (EC) and observed by the designer, which facilitated understanding user feedback. Through the use of the "think-aloud" technique, a cognitive interviewing method, participants were asked to perform a task within the app while the researcher observed [42]. Users were asked to describe their use of the app as they performed the tasks, including what they liked or disliked, what was confusing, and whether they would change anything. ...
Background
The school-age years, approximately ages 7 through 11, represent a natural transition when children begin assuming some responsibility for their asthma management. Previously, we designed a theoretically derived, tailored parent–child shared asthma management mobile health app prototype, Improving Asthma Care Together (IMPACT).
Objective
The purpose of this study was to use human-centered design (HCD) to iteratively refine IMPACT to optimize user experience and incorporate evidence-based longitudinal engagement strategies.
Methods
This study used a mixed methods design from December 2019 to April 2021. Our app refinement used the HCD process of research, ideation, design, evaluation, and implementation, including 6 cycles of design and evaluation. The design and evaluation cycles focused on core app functionality, child engagement, and overall refinement. Evaluation with parent–child dyads entailed in-person and remote concept testing and usability testing sessions, after which rapid cycle thematic analyses identified key insights that informed future design refinement.
Results
Twelve parent–child dyads enrolled in at least one round of this study. Eight of the 12 child participants were male with a mean age of 9.9 (SD 1.6) years and all parent participants were female. Throughout evaluation cycles, dyads selected preferred app layouts, gamification concepts, and overall features with a final design prototype emerging for full-scale development and implementation.
Conclusions
A theoretically derived, evidence-based shared asthma management app was co-designed with end users to address real-world pain points and priorities. An 8-week pilot study testing app feasibility, acceptability, and preliminary efficacy is forthcoming.
... Usability sessions were moderated by the UX researcher (EC) and observed by the designer, which facilitated understanding user feedback. Through the use of the "think-aloud" technique, a cognitive interviewing method, participants were asked to perform a task within the app while the researcher observed [42]. Users were asked to describe their use of the app as they performed the tasks, including what they liked or disliked, what was confusing, and whether they would change anything. ...
Background: The school-age years, approximately ages seven through eleven, represent a natural transition when children begin assuming some responsibility for their asthma management. Previously, we designed a theoretically derived, tailored parent-child shared asthma management mHealth application (app) prototype, Improving Asthma Care Together (IMPACT).
Objective: The purpose of this study was to use human centered design (HCD) to iteratively refine IMPACT to optimize user experience and incorporate evidence-based longitudinal engagement strategies.
Methods: This study used a mixed methods design from December 2019 through April 2021. Our app refinement used the HCD process of research, ideation, design, evaluation, and implementation, including six cycles of design and evaluation. The design and evaluation cycles focused on core app functionality, child engagement, and overall refinement. Evaluation with parent-child dyads entailed in-person and remote concept testing and usability testing sessions, after which rapid cycle thematic analyses identified key insights that informed future design refinement.
Results: Twelve parent-child dyads enrolled in at least one round of this study. Eight of the 12 child participants were male with a mean age of 9.9 + 1.6 years and all parent participants were female. Throughout evaluation cycles, dyads selected preferred app layouts, gamification concepts, and overall features with a final design prototype emerging for full-scale development and implementation.
Conclusions: A theoretically derived, evidence-based shared asthma management app was co-designed with end users to address real-world pain points and priorities. An eight-week pilot testing app feasibility, acceptability, and preliminary efficacy is forthcoming.
... In order to answer our research question, we used the think-aloud method. Although the think-aloud method has some shortcomings, it is widely seen as a reliable method to gain insight into thinking and reasoning processes (e. g., Charters, 2003;Pressley & Afflerbach, 1995;van Someren et al., 1994;Wineburg 1991). ...
The concept of historical significance is seen as a key concept of historical reasoning. Assigning significance is based on criteria and related to the identity of who assigns significance. However, little is known about reasoning-, reading-, and writing processes when students attribute significance. The aim of this study is to investigate how students and experienced history teachers with a master’s degree reason, read, and write about historical significance while thinking aloud. We analyzed the think-aloud protocols of twelve 10th-grade students and four history teachers on reasoning, reading, and writing processes. While thinking aloud, participants read two contrasting accounts after which they wrote an argumentative text about the historical significance of Christopher Columbus. Analysis of participants’ think-aloud protocols and their written texts showed that students did not recognize historical accounts as perspectives—influenced by the historical context. In contrast, teachers looked for the authors’ judgement, evidence, and context. In addition, students’ limited use of metaknowledge regarding texts and the concept of historical significance hampered them. These out-comes provide direction for teaching reasoning, reading, and writing with respect to historical significance.
... After a brief introduction to our research project, we presented the prototype to the participants and asked them to express any experience and thought following the think-aloud method [56]. We did not explain the features of the prototype but wanted to see how users get along themselves, how intuitive the approach is, and hear what participants think while exploring the prototype. ...
... As part of the development stage, think-aloud interviews [35] were carried out with 19 participants with acne using the draft intervention to gather feedback and further modify the intervention. Participants were recruited through mail-out from primary care practices, opportunistic sampling using posters, and advertising via social media. ...
Background: acne is a common skin condition that is most prevalent in young people. It can have a substantial impact on the quality of life, which can be minimized with the appropriate use of topical treatments. Nonadherence to topical treatments for acne is common and often leads to treatment failure.
Objective: the aim of this study is to develop a web-based behavioral intervention to support the self-management of acne and to assess the feasibility of recruitment, retention, and engagement of users with the intervention.
Methods: the intervention was developed iteratively using the LifeGuide software and following the person-based approach for intervention development. The target behavior was appropriate use of topical treatments. Barriers and facilitators identified from the qualitative research and evidence from the wider literature were used to identify techniques to improve and promote their use. Young people with acne aged 14-25 years who had received treatment for acne in the past 6 months were invited to participate through mail-out from primary care practices in the South of England in a parallel, unblinded randomized trial. Participants were automatically randomized using a computer-generated algorithm to usual care or to usual care plus access to the web-based intervention. Usage data was collected, and a series of questionnaires, including the primary outcome measure for skin-specific quality of life (Skindex-16), were collected at baseline and at the 4- and 6-week follow-ups.
Results: a total of 1193 participants were invited, and 53 young people with acne were randomized to usual care (27/53, 51%) or usual care plus intervention (26/53, 49%). The response rate for the primary outcome measure (Skindex-16) was 87% at 4 weeks, 6 weeks, and at both time points. The estimate of mean scores between groups (with 95% CI) using linear regression showed a trend in the direction of benefit for the web-based intervention group in the primary outcome measure (Skindex-16) and secondary measures (Patient Health Questionnaire-4 and the Problematic Experiences of Therapy Scale). Intervention usage data showed high uptake of the core module in the usual care plus web-based intervention group, with 88% (23/26) of participants completing the module. Uptake of the optional modules was low, with less than half visiting each (myth-busting quiz: 27%; living with spots or acne: 42%; oral antibiotics: 19%; what are spots or acne: 27%; other treatments: 27%; talking to your general practitioner: 12%).
Conclusions: this study demonstrated the feasibility of delivering a trial of a web-based intervention to support self-management in young people with acne. Additional work is needed before a full definitive trial, including enhancing engagement with the intervention, recruitment, and follow-up rates.
... Following this general interview, we conducted a think-aloud interview to garner additional information about ID coaching practice, strategies, and approach (Charters, 2003;Chipman et al., 2000). Think-aloud activities involve a participant solving a problem and making a decision by verbalizing their thoughts aloud while performing a task. ...
Background/Context
School leaders are central to state and district human-capital reforms (HCRs), yet they are rarely equipped with the skills to implement new evaluation, professional development, and personnel data systems. Although districts increasingly offer principals coaching and training, there has been limited empirical work on how these supports influence principals’ HCR-related practices.
Purpose
Drawing on a two-year, mixed-methods study in the Los Angeles Unified School District (LAUSD), this article examines the role of principal supervisors in HCRs. We ask: What role did principal supervisors (Instructional Directors [IDs]) play in the implementation of human-capital reforms? What did high-quality coaching on the part of IDs look like in this context?
Research Design
Our two-part analysis draws upon survey and interview data. First, we conducted descriptive analyses and significance testing using principal and ID survey data to examine the correlations among principals’ ratings of ID coaching quality, ID coaching practices, and principals’ implementation of HCRs. Second, we conducted in-depth interviews, using a think-aloud protocol, with two sets of IDs—those consistently highly-rated and those with mixed ratings—who were identified using principals’ reports of coaching quality. Following interview coding, we created various case-ordered metamatrix displays to analyze our qualitative data in order to identify patterns in coaching strategy and approach across IDs, content, and contexts.
Findings
First, our survey data indicate that receiving high-quality coaching from IDs is correlated with stronger principal support for and implementation of HCRs. Our survey findings further illustrate that IDs support a wide range of principals’ HCR activities. Second, our think-aloud interviews with case IDs demonstrate that coaching strategy and approach vary between consistently highly-rated and mixed-rated coaches: Consistently highly-rated IDs emphasize the importance of engaging in, or defining HCR problems as, joint work alongside principals, while mixed-rated IDs often emphasize the use of tools to guide principal improvement. We find that, on the whole, the consistently highly-rated IDs in our sample employ a nondirective approach to coaching more often than mixed-rated coaches.
Conclusions
These findings contribute to a growing literature on the crucial role of principal supervisors as coaches to improve principals’ instructional leadership and policy implementation. While exploratory, this study offers the first steps toward building greater evidence of the connections between high-quality coaching and policy implementation, and it may have implications for the design and implementation of professional development for principal supervisors and the selection and placement of supervisors with principals.
... A convenience sample of intervention participants was invited to participate in an accompanied shopping trip whereby the lead author observed them doing their usual supermarket shopping, using the think-aloud method that attempts to access and record participants' inner speech, the transformation of thought processes into words [45]. The think-aloud method has been widely used in the design and evaluation of digital interventions [46,47] and to explore consumers' food choice behaviors [48][49][50][51]. ...
BACKGROUND
A high-salt diet is a risk factor for hypertension and cardiovascular disease; therefore, reducing dietary salt intake is a key part of prevention strategies. There are few effective salt reduction interventions suitable for delivery in the primary care setting, where the majority of the management and diagnosis of hypertension occurs.
OBJECTIVE
The aim of this study is to assess the feasibility of a complex behavioral intervention to lower salt intake in people with elevated blood pressure and test the trial procedures for a randomized controlled trial to investigate the intervention’s effectiveness.
METHODS
This feasibility study was an unblinded, randomized controlled trial of a mobile health intervention for salt reduction versus an advice leaflet (control). The intervention was developed using the Behavior Change Wheel and comprised individualized, brief advice from a health care professional with the use of the SaltSwap app. Participants with an elevated blood pressure recorded in the clinic were recruited through primary care practices in the United Kingdom. Primary outcomes assessed the feasibility of progression to a larger trial, including follow-up attendance, fidelity of intervention delivery, and app use. Secondary outcomes were objectively assessed using changes in salt intake (measured via 24-hour urine collection), salt content of purchased foods, and blood pressure. Qualitative outcomes were assessed using the think-aloud method, and the process outcomes were evaluated.
RESULTS
A total of 47 participants were randomized. All progression criteria were met: follow-up attendance (45/47, 96%), intervention fidelity (25/31, 81%), and app use (27/31, 87%). There was no evidence that the intervention significantly reduced the salt content of purchased foods, salt intake, or blood pressure; however, this feasibility study was not powered to detect changes in secondary outcomes. Process and qualitative outcomes demonstrated that the trial design was feasible and the intervention was acceptable to both individuals and practitioners and positively influenced salt intake behaviors.
CONCLUSIONS
The intervention was acceptable and feasible to deliver within primary care; the trial procedures were practicable, and there was sufficient signal of potential efficacy to change salt intake. With some improvements to the intervention app, a larger trial to assess intervention effectiveness for reducing salt intake and blood pressure is warranted.
CLINICALTRIAL
International Standard Randomized Controlled Trial Number (ISRCTN): 20910962; https://www.isrctn.com/ISRCTN20910962
... A convenience sample of intervention participants was invited to participate in an accompanied shopping trip whereby the lead author observed them doing their usual supermarket shopping, using the think-aloud method that attempts to access and record participants' inner speech, the transformation of thought processes into words [45]. The think-aloud method has been widely used in the design and evaluation of digital interventions [46,47] and to explore consumers' food choice behaviors [48][49][50][51]. ...
Background
A high-salt diet is a risk factor for hypertension and cardiovascular disease; therefore, reducing dietary salt intake is a key part of prevention strategies. There are few effective salt reduction interventions suitable for delivery in the primary care setting, where the majority of the management and diagnosis of hypertension occurs.
Objective
The aim of this study is to assess the feasibility of a complex behavioral intervention to lower salt intake in people with elevated blood pressure and test the trial procedures for a randomized controlled trial to investigate the intervention’s effectiveness.
Methods
This feasibility study was an unblinded, randomized controlled trial of a mobile health intervention for salt reduction versus an advice leaflet (control). The intervention was developed using the Behavior Change Wheel and comprised individualized, brief advice from a health care professional with the use of the SaltSwap app. Participants with an elevated blood pressure recorded in the clinic were recruited through primary care practices in the United Kingdom. Primary outcomes assessed the feasibility of progression to a larger trial, including follow-up attendance, fidelity of intervention delivery, and app use. Secondary outcomes were objectively assessed using changes in salt intake (measured via 24-hour urine collection), salt content of purchased foods, and blood pressure. Qualitative outcomes were assessed using the think-aloud method, and the process outcomes were evaluated.
Results
A total of 47 participants were randomized. All progression criteria were met: follow-up attendance (45/47, 96%), intervention fidelity (25/31, 81%), and app use (27/31, 87%). There was no evidence that the intervention significantly reduced the salt content of purchased foods, salt intake, or blood pressure; however, this feasibility study was not powered to detect changes in secondary outcomes. Process and qualitative outcomes demonstrated that the trial design was feasible and the intervention was acceptable to both individuals and practitioners and positively influenced salt intake behaviors.
Conclusions
The intervention was acceptable and feasible to deliver within primary care; the trial procedures were practicable, and there was sufficient signal of potential efficacy to change salt intake. With some improvements to the intervention app, a larger trial to assess intervention effectiveness for reducing salt intake and blood pressure is warranted.
Trial Registration
International Standard Randomized Controlled Trial Number (ISRCTN): 20910962; https://www.isrctn.com/ISRCTN20910962
... In order to answer our research question, we used the think-aloud method. Although the think-aloud method has some shortcomings, it is widely seen as a reliable method to gain insight into thinking and reasoning processes (e. g., Charters, 2003;Pressley & Afflerbach, 1995;van Someren et al., 1994;Wineburg 1991). ...
... Both the questionnaire and sleep diary were pre-tested among six parents representative of the study population. The comprehensibility of the study materials was pre-tested using the "think aloud method" (Charters, 2003) and necessary changes were made accordingly. The pre-test showed that the materials were well-understood by parents. ...
The aim of the present study was to explore potential factors of inadequate sleep health (i.e. sleep duration, quality, and timing) of school‐aged children. Data were collected among 382 primary school children (aged 4–13 years) and their parents. Personal characteristics (i.e. age, sex), individual lifestyle behaviours (i.e. screen use, sleep hygiene behaviour), social and community factors (i.e. parental sleep‐related practices, parental barriers, perceived ethnicity), and living conditions (i.e. parental educational level, sleep environment) were assessed with a parental questionnaire. Sleep duration, quality, and timing were assessed with a sleep diary. Associations were analysed using linear mixed models and logistic regression analyses. In total, 332 children, with a mean (range) age of 7.5 (4–13) years, were included in the analyses. The mean sleep duration was 632 min/night, the mean sleep quality score was 40, on a scale from 10 to 50, and 25% had a bedtime that varied >40 min between weekdays. Factors negatively associated with children’s sleep health included older age, perceived non‐Dutch cultural background, lower parental pre‐sleep emotional support, the parental barrier to get their child to bed on time when siblings have a later bedtime, high parental educational level, sleeping in a darkened bedroom, and being brought to bed after falling asleep. On average, children in the present study had adequate sleep health. The factors found to be associated with children’s sleep health are useful for future healthy sleep research and intervention development.
... Some participants may be uncomfortable sharing their opinions whilst being prompted and recorded, although Think-Aloud has been shown to be a robust method in other studies. 30,31 Future research repeating the current study could assess the pre-and post-participant user experience with education level as covariate. Regardless, incorporating user testing of enhancements to the website architecture, design and content should be a routine part of the website maintenance. ...
Background: With increasing accessibility of the World Wide Web, patients are using it to obtain patient education materials (PEM). With this in mind, our group (surgeons, radiation oncologists, medical oncologists, nursing, allied health professionals and academic researchers) developed a comprehensive information resource for patients with head and neck cancer (HNC), the Head and Neck Cancer Australia, formerly Beyond Five website. The aim of this study was to determine patient usability of the Head and Neck Cancer Australia website. Methods: Usability testing (Cognitive walkthrough & Think-Aloud) in 18 patients treated for HNC was undertaken at a Cancer centre (anonymous for reviewers). Results: The tasks rated easiest by patients were finding information on psychological well-being (mean time spent: 66 s, range: 10–565), health and well-being effecting quality of life (mean time spent: 36 s, range 9–117) and carer information (mean time spent: 10 s, range 3–35). Patients indicated the website contained a lot of information, covers most topics, was a trusted source of information and a springboard to other information. Conclusions: The Head and Neck Cancer Australia website provides a wide range of information and support in multiple formats available to HNC patients. Further refinements in design, navigation and website instructions are needed to allow effective patient interaction.
... This version was piloted with two representatives of the two municipal head organizations in both federal states, the "Schleswig-Holsteinischer Gemeindetag" and the "Gemeindetag Baden-Württemberg". Feedback was collected using the think-aloud technique [24]. Main questions were understandability and clearness of the items. ...
Background
Telemedicine offers additional ways of delivering medical care, e.g., in primary care in rural areas. During the last decades, projects including telemedicine are being implemented worldwide. However, implementation of telemedicine is in some countries, e.g., Germany somewhat slower compared to northern European countries. One important part of successful implementation is to include the citizen perspective. The aims of this study were to explore the perception of representatives of the local government regarding telemedicine in the context of a perceived GP shortage and to tailor future telemedicine offers according to these perceived needs.
Methods
Considering the multidisciplinary assessment suggested by the Model for Assessment of Telemedicine a questionnaire with 19 questions was developed by identifying determinants of telemedicine out the literature. After pre-testing, the questionnaire was sent to all 2199 mayors from the federal states of Schleswig-Holstein (North Germany) and Baden-Württemberg (South Germany) as representatives of the citizens (cross- sectional study; full population survey). The final questionnaire contained sections for socio-demographic data, telemedicine and perceived GP shortage. All responses from November 2018 until 2019 were included and analyzed descriptively.
Results
The response rate was 32% ( N = 699), of which 605 were included in the analysis. A majority of the participants stated they live in a rural area and 46% were in the office for up to 8 years. The mayors had predominantly a positive perception about telemedicine (60%) and 76% of them stated, their community would benefit from telemedicine. A GP shortage was reported by 39% of the participants. The highest risk of telemedicine was seen in misdiagnosing. In case of an emergency situation 291 (45%) of the participants considered data privacy as not as relevant. Mayors from a community with a perceived GP shortage had a more negative perception regarding telemedicine.
Conclusion
The acceptance of telemedicine is rapidly rising compared to former studies. Communities with a perceived GP shortage had a more negative perception. Barriers like data security concerns were seen as less important in case of an emergency. The highest risk of telemedicine was seen in misdiagnosing. These findings need to be considered in designing future telemedicine offers.
... Youth showcased their films at a Summer 2020 screening hosted by Northwestern University's Block Museum, which also included a Q&A with the virtual audience moderated by a PhD student and member of the design team (https://vimeo.com/showcase/7260847). To understand the relationship between students' filmic design choices (e.g., use of B-roll, voice-over, montage sequences) and their sensemaking, we conducted 90-minute think-aloud interviews with student documentary teams (Charters, 2003;Ericsson & Simon, 1980). Prior to these interviews, youth re-watched their films and identified specific moments (marked with time codes) they felt were especially representative of the essence of the film (see Appendix A). ...
... For instance, the assessments of improvements in understanding and insights are highly subjective and need to rely on verbal feedback. Examples of qualitative methods to explore the user experience include the Think Aloud technique, which allows gathering users' comments and impressions during the tests (Charters 2003). The Bipolar Laddering method is also useful to quantify users' insights and their impact, by evaluating the positive and negative aspects found in the visualisation and rating them according to their importance (for positive aspects) and severity (for negative aspects) (Pifarré and Tomico 2007). ...
Visualisations are often the entry point to information that supports stakeholders’ decision- and policy-making processes. Visual displays can employ either static, dynamic or interactive formats as well as various types of representations and visual encodings, which differently affect the attention, recognition and working memory of users. Despite being well-suited for expert audiences, current climate data visualisations need to be further improved to make communication of climate information more inclusive for broader audiences, including people with disabilities. However, the lack of evidence-based guidelines and tools makes the creation of accessible visualisations challenging, potentially leading to misunderstanding and misuse of climate information by users. Taking stock of visualisation challenges identified in a workshop by climate service providers, we review good practices commonly applied by other visualisation-related disciplines strongly based on users’ needs that could be applied to the climate services context. We show how lessons learned in the fields of user experience, data visualisation, graphic design and psychology make useful recommendations for the development of more effective climate service visualisations. This includes applying a user-centred design approach, using interaction in a suitable way in visualisations, paying attention to information architecture or selecting the right type of representation and visual encoding. The recommendations proposed here can help climate service providers reduce users’ cognitive load and improve their overall experience when using a service. These recommendations can be useful for the development of the next generation of climate services, increasing their usability while ensuring that their visual components are inclusive and do not leave anyone behind.
... Think-aloud method A think-aloud approach, in which advisors speak their thoughts aloud while performing a task, will be used to collect feedback on patient trial information documents. 16 The documents will be provided at the beginning of the meeting, and the patient advisors will be asked to verbalise their thoughts while reading it aloud. An assistant will take notes to contribute to the digitally recorded material. ...
Introduction:
A patient and public involvement (PPI) project will be embedded within the SALuBRITY pilot trial, a two parallel group, double sham controlled, randomised clinical trial. The study aims to compare the effectiveness of spinal manual therapy and corticosteroid nerve root injections, two methods commonly used to treat patients with lumbar radiculopathy. We aim to gather patients' and clinicians' perspectives and involve them in decisions related to the research question and objectives, proposed trial recruitment processes and methods, and proposed outcome measures.
Methods and analysis:
A small group of patients with lived experience of lumbar radiculopathy and primary care clinicians with experience in the treatment of patients with lumbar radiculopathy are involved. An initial kickoff event will prepare and empower the advisors for involvement in the project, followed by semistructured patient group and one-on-one clinician interviews. We will follow the Critical Outcomes of Research Engagement framework for assessing the impact of patient engagement in research. We will summarise and feedback PPI content to the patient and clinician advisors during a member-checking process to ensure accurate interpretation of patient and clinician inputs. Inductive and deductive thematic analysis will be used for the qualitative analysis of the interviews. Two surveys will be completed at different points along the trial to track the advisors' and researchers' experiences over the course of the PPI project. Any modifications to the SALuBRITY trial methods due to PPI inputs will be thoroughly documented and recorded in an impact log.
Ethics and dissemination:
The independent research ethics committee of Canton Zurich confirmed that ethical approval for this PPI subproject was not required. PPI results will be disseminated in a peer-reviewed journal and presented at conferences.
... Section 3.2). Drawing from the thinking-aloud method [48], we asked participants to comment on the anonymized app comparison matrix. This increased the trustworthiness (credibility) of our literature analysis in terms of completeness, relevance, and categorization/labelling [49]. ...
Information systems (IS) play a central role in promoting corporate sustainability and pro-environmental behavior. This study explores the use of mobile apps for fostering sustainability-oriented corporate culture. It accentuates issues relating to sustainability-oriented corporate culture (RQ1), app meta-requirements as a strategic approach to addressing these issues (RQ2), as well as design and implementation principles (RQ3). Referring to the literature on sustainability apps, gamification, and nudging, our qualitative research design combines an analysis of four corporate apps intending to promote sustainable behavior and expert interviews (Grounded Theory). The proposed framework supports the planning, realization, and monitoring of this targeted app use. Single cultural dimensions inform seven issues in culture development. Five meta-requirements address this multi-dimensionality, challenges provided by new digital working environments, and principles of Green IT. Four design principles support, extend, and integrate current knowledge on app features, nudging, and gamification. Five variables determine the app intervention and maturity level. We conclude that this targeted app use should intend to foster the sustainability orientation within all dimensions of corporate culture instead of being limited to promoting sustainable behavior.
... Before starting to use the training platform, it will be explained to the participants that while they are carrying out the tasks, they should speak out loud everything they would say to themselves if they had to be silent. 35 ...
Introduction
Vaccination is a fundamental intervention in disease prevention; therefore, the advice and recommendations of health professionals have a major influence on the population’s decision to be vaccinated or not. Professionals must have sufficient competencies to carry out their work and recommend vaccination with evidence-based knowledge. The aim is to design and validate a strategy to improve professional competencies in vaccination to positively influence adherence and increase vaccination rates in the population.
Methods and analysis
Training will be designed based on evidence and previous studies and piloted with healthcare providers. To test changes in knowledge, a pretest and post-test will be conducted. To test feasibility, a think-aloud method will be used with participants and triangulated with focus groups using SWOT (strengths, weaknesses, opportunities and threats) analysis. Transfer will be measured using the questionnaire ‘factors for the indirect evaluation of transfer’ and an efficacy questionnaire 1½ months later; for satisfaction, an ad hoc questionnaire will be used. A summative approach will be used for the analysis of the focus groups and descriptive and bivariate statistics for the questionnaires.
Ethics and dissemination
This study was approved by the Andalusian Research Ethics Committee, Spain (approval number: 0524-N-20). The results will be made available to the public at journal publications and scientific conferences.
... To examine response process validity (Padilla & Benitez, 2014), two researchers (LCF and ALP) conducted cognitive interviews (Charters, 2003) with three faculty who precept learners on IP clinical teams. Raters were instructed to think aloud as they rated each video. ...
Collaborative knowledge construction (KC) is an important process in interprofessional learning and a logical assessment target. A tool supporting the formative evaluation of KC behaviors ideally would be: 1) applicable to interprofessional teams of learners in clinical contexts; 2) informed by contemporary learning frameworks; 3) feasible and useful. No existing assessment tool meets these criteria. This paper describes the development and preliminary validity evidence for a Tool for Observing Construction of Knowledge in Interprofessional teams (TOCK-IP). Following literature review and needs assessment, the TOCK-IP was drafted based upon Gunawardena's five-phase KC model. Educational expert review established content validity. Response process and internal structure validity, feasibility, and utility were assessed through step-wise evaluation. Faculty raters applied the tool to four videos of simulated interactions between health professions learners. Faculty ratings were compared to expert consensus ratings. Thematic analysis of post-rating survey and debrief allowed assessment of feasibility and utility. Across videos, faculty raters' agreement was fair (n = 25; Fleiss' kappa = 0.40, <0.001). Excellent agreement (95%) was found for raters' scores compared to consensus rating. Faculty supported tool feasibility and utility. The TOCK-IP meets the three criteria for evaluating team-level KC and offers a progression roadmap to help learners move toward collaborative learning.
... This is an argument in favor of posing questions aloud. In connection with this, it is worth mentioning that talk-aloud techniques (describing one's cognitive process out loud while conducting a task) have been in use in psychological research for a while (Charters, 2003). Furthermore, to enhance either the interviewer or the interviewee role, different pronouns may be used. ...
This article presents and discusses the method of micro-phenomenological self-inquiry. Micro-phenomenology is usually performed with two persons, one interviewer and one interviewee. Micro-phenomenological self-inquiry consists of one person investigating their own experience. The different aspects of the regular micro-phenomenological interview are reviewed in relation to the process of self-inquiry. Examples of recent studies that apply micro-phenomenological self-inquiry are presented. Finally, there is a discussion of some methodological problems and objections. Advantages and disadvantages of micro-phenomenological self-inquiry are considered. Suggestions are given for what kind of research and research contexts that micro-phenomenological self-inquiry may be particularly suitable for.
Background:
Accurate interpretation of a 12-lead electrocardiogram (ECG) demands high levels of skill and expertise. Early training in medical school plays an important role in building the ECG interpretation skill. Thus, understanding how medical students perform the task of interpretation is important for improving this skill.
Objective:
We aimed to use eye tracking as a tool to research how eye fixation can be used to gain a deeper understanding of how medical students interpret ECGs.
Methods:
In total, 16 medical students were recruited to interpret 10 different ECGs each. Their eye movements were recorded using an eye tracker. Fixation heatmaps of where the students looked were generated from the collected data set. Statistical analysis was conducted on the fixation count and duration using the Mann-Whitney U test and the Kruskal-Wallis test.
Results:
The average percentage of correct interpretations was 55.63%, with an SD of 4.63%. After analyzing the average fixation duration, we found that medical students study the three lower leads (rhythm strips) the most using a top-down approach: lead II (mean=2727 ms, SD=456), followed by leads V1 (mean=1476 ms, SD=320) and V5 (mean=1301 ms, SD=236). We also found that medical students develop a personal system of interpretation that adapts to the nature and complexity of the diagnosis. In addition, we found that medical students consider some leads as their guiding point toward finding a hint leading to the correct interpretation.
Conclusions:
The use of eye tracking successfully provides a quantitative explanation of how medical students learn to interpret a 12-lead ECG.
Background
Diet and fitness apps are often promoted in university and college settings and touted as a means to improve health with little attention given to unanticipated negative effects, especially among those at risk for or with eating disorders.
Aims
Few researchers have studied how these apps affect women with eating disorders in university and college settings. This research investigates the unintended negative consequences of engaging with these tools.
Method
Data collection sessions comprised three components conducted with 24 participants: survey (demographic and eating disorder symptoms), think-aloud exercise and semi-structured interview. Thematic analysis was used to analyse data.
Results
Participants reported that diet and fitness apps trigger and exacerbate symptoms by focusing heavily on quantification, promoting overuse and providing certain types of feedback. Eight themes of negative consequences emerged: fixation on numbers, rigid diet, obsession, app dependency, high sense of achievement, extreme negative emotions, motivation from ‘negative’ messages, and excess competition. Although these themes were common when users’ focus was to lose weight or eat less, they were also prevalent when users wanted to focus explicitly on eating disorder recovery.
Conclusions
Unintended negative consequences are linked to the quantified self movement, conception of appropriate usage, and visual cues and feedback. This paper critically examines diet and fitness app design and discusses implications for designers, educators and clinicians. Ultimately, this research emphasises the need for a fundamental shift in how diet and fitness apps promote health, with mental health at the forefront.
Background:
There is a lack of research regarding factors promoting recovery during the workday and effective interventions.
Objective:
To evaluate how different intervention activities may impact employees' experiences of recovery at the workplace.
Methods:
Customized intervention activities based on qualitative results and a participatory approach were integrated among the employees at six primary health care centres (PHCCs; n = 166) during one year. Recovery and workplace factors were measured with a questionnaire at the start and end of intervention, and also in a control group (15 PHCCs; n = 328). Group differences were tested (Chi-2) and explanatory factors compared by logistic regression models.
Results:
The proportion of employees reporting workday recovery increased in the intervention group (19.9%to 29.1%; p = 0.01), whereas the control group showed no significant change. Recovery was explained by self-reflection and reflection with co-workers. After intervention, having influence on work situation, energy-building experience, and opportunity for laughter also contributed significantly to recovery.
Conclusions:
The results contribute to work recovery research by confirming that a customized intervention may have an impact on employees' recovery experiences. The study showed that considering the factors of reflection, influence, and companionship can positively impact workplace recovery.
Androgen deprivation therapy (ADT) is a hormone treatment for prostate cancer and is linked to altered cognition, including memory changes, which can have detrimental effects on quality of life (QOL). The aim of this research was to develop an intervention to support men with memory changes associated with ADT. The Intervention Mapping framework guided intervention development through a series of rigorous steps. A self-management booklet called MEMORY MANager was developed, which incorporated education, self-assessment, compensatory strategies and techniques to improve mood. It was found to be an acceptable means of supporting men. This booklet could be valuable to healthcare professionals and patients as a means of improving QOL among men who undergo ADT. This is the first reported self-management intervention developed for managing cognitive changes in men who undergo ADT. Although preliminary, findings appear promising. More research is needed to evaluate effectiveness of the intervention and its implementation.
Digital media initiated a paradigm shift of the doctor-patient relationship in which prospective patients "shop" for physicians as they do other goods and services: using search engines' results, aggregated review websites, and star ratings as cues. These are processed centrally or peripherally in accordance with the Elaboration Likelihood Model (ELM). Electronic word of mouth (EWOM) in conjunction with a pediatrician's digital brand management strategy impacts practice revenue through patient recruitment and retention. This exploratory study found that a pediatrician's placement on a review website and overall digital brand image leads to increased or decreased trust in expertise, competence, and likability.
The thesis “If You Waver, You’re Tarnished” – Danish Oral Exams from a Student Perspective explores Danish upper secondary students’ preparatory processes for and conceptions of oral exams. It takes the form of an investigation of central policy documents as well as and ethnographic study, following the students’ transition from the contexts of teaching and learning in the classrooms through their independent preparation for the oral exams asking for their reflections on the criteria for a successful performance at the exams. The hitherto under-researched Danish oral exams are high stakes exam forms practiced since 1848, which students complete after classroom teaching has ended. The Danish Ministry of Education will in late May shortly before the exam season in June decide which two or three subjects each individual student is to attend. Oral exam formats differ, but a classic is the “pick-a-question”-model where students have a short time (from 24 minutes to 24 hours) to prepare. The student’s performance results in a grade and immediate oral feedback. The total score of grades is an administrative indicator for students’ future potential free choice of further education. Since oral exams are rhetorical situations of great importance, it can be assumed that students are intensely engaged with them. A basic assumption of the thesis is that students’ perception of what they expect to be assessed on will influence their studying and their presentations at the exams.
Oral exams from a student perspective have only been researched sporadically in Denmark (Borgnakke, 1996; Miller, 2004; Nissen, 2019) and only scattered internationally (Huxham et al., 2012; Joughin, 2003; Kvifte, 2011). Since the Danish-German didactical tradition leaves assessment of learning little theoretical attention, the thesis finds theoretical inspiration from related fields of study.
The research questions that shape the dissertation are: How do upper secondary school students navigate the Danish oral examination system? What is assessed in the oral exams according to central policy documents? How do students investigate and voice what is assessed in oral exams? And how do students perceive the (Bachtinian) addressees of the oral exam situation when entering the exam? What considerations do these finding give rise to – for future practice of oral examinations in Danish upper secondary school and for Danish exam research? The thesis answers these questions in three articles and an extended abstract by investigating policy documents and fieldwork material employing a student perspective. The fieldwork focused closely on four 18-20-year-old case-students from three classrooms (80 students) in two schools across 10 subjects taught by 18 teachers in the spring 2018 preparing for their oral presentations. Eight oral exams were observed.
The article Construct, Generalization and Luck in Danish Oral Examinations in Upper Secondary School (Acta Didactica Norden, 14(3), 21.) uses Kane’s internationally acknowledged argument-based approach to validation of assessment in education (Kane, 2006) in analysis of central policy documents to identify what information students may find on what is assessed in oral exams. The documents are rather vague about the intended construct. Since students are only tested in sampled subjects that the Ministry of Education will decide on, implicit warrants of the exam system must be that there is basis for extensive generalization from one score interpretation to potential other performances across the curricula. The oral exam performance in one topic is generalized to other topics within the subject, and the same performance is generalized to other subjects, that students are not tested in.
194
Since the official policy documents have only vague expression of what is to be assessed, the two following articles investigate how students perceive the criteria of oral exam. Using Bitzer’s theory of the rhetorical situation (Bitzer, 1968) and Moje’s concept of navigation (Moje, 2013), the article ”Oral Exams Are an Art Form” – Danish Upper Secondary Students Attending Oral Exams investigates three case-students’ perceived fitting responses for the oral exam in the subject of history and English and how they identify these responses. The analysis finds that formulations of a good exam performance in class differs from students’ formulations. One case-student navigates the policy documents as well as listens to the stated criteria in the classroom context and expresses numerous aspects of a fitting response, while another case-student seem to navigate primarily by observing classroom interactions and feels criteria rather than expresses them. The third case-student navigates via classroom interaction in search of consolidated models of the expected fitting response at the oral exam. Since such models are invisible to her in class, she is frustrated and left without a sense of a fitting response. Even though this was never stated in class or in policy documents, it turns out that not stepping over the line to competing disciplinary fields is an important element of a fitting response at the oral exam. Thereby, having strong disciplinary skills in a competing field of study could be a potential drawback at the oral exam.
Using Bachtin’s dialogical concepts of addressees and superaddressees, along with positioning theory (Davies & Harré, 2014), the article ”Cracking the Teacher’s Code” – Students’ Perceived Addressees Before Oral Exams investigates students’ expectations of the exam addressees since they seem focused on cracking the teacher’s code instead of delivering the curriculum requirement of independent critical interpretations of text material. The article analyzes 13 incidents where students and teachers negotiate oral exams asking questions and giving advice on what to do and prepare for before the oral exam. One key incident exam introduction is chosen for close analysis in order to show how interactions in class create a history of utterances that students use when they anticipate and imagine the oral exam dialogue. Results show students withdrawing from independent analysis and disciplinary traditions at the exam, since classroom dialogues indicated that ‘alternative’ interpretations were considered a confrontation with the teacher-assessor. Consequently, students’ interpretations are not necessarily available for assessment at oral exam.
The uncertainty about what oral exams are intended to assess seems to be a consistent finding in policy documents as well as in the students’ experience. The three articles together show that the extensive warrants of generalizations between disciplinary subjects (horizontal generalization) and from a specific subject to the global intentions for upper secondary school students’ learning (vertical generalization) found in the policy documents are questionable in practice. Students navigations show that oral exams are perceived as highly contextual and depending on the preferences of the specific teacher performing the assessment. Students generate their understanding of exam criteria in everyday teaching, why oral exams are not just washing back on the interactions in class, the same interactions seem to wash exam criteria forward in the students’ perception.
A central discussion in the extended abstract is the import of the Anglo-Saxon concept of the construct and how is corresponds with the Danish-German didactical tradition. Since discussions about assessment is rare in Danish research and public debate, the argument-based approach to investigations of exams offer an outsider’s view on a long-standing Danish exam form. It is suggested that assessment theory can inspire future research and exam design activities: Oral exams are designs, but since oral exams are dialogic interactions between student and assessor,
195
this dialogue can be practiced differently in the attempt to sync theoretical approaches to assessment and learning.
Implications of the research findings are aimed at the macro level of policy makers and locally to teachers practicing oral assessment. On a macro level, it is recommended to offer argumentation as to why Danish oral exams are valid assessment of students’ competencies and to reduce the elements of randomness in the assessed disciplines for the individual student. Support or training could be offered to raters. On a local level it is suggested to focus on creating interpretative communities in order to agree on score interpretations and to focus on how everyday dialogic practices shape students’ expectation of exam criteria. The student-perspective on assessment would suggest letting students participate in discussions of what is assessed in oral exams.
Thus, the thesis contributes key findings and perspectives to the hitherto under-researched field of assessment in education, in a Danish context as well as internationally.
Background
The Danish version of Health Care Students’ Attitudes towards Addressing Sexual Health (SA-SH-D) is a questionnaire for measuring the attitudes of health care professional students’ towards addressing sexual health in their future professional practice and care.
Aim
To assess content validity and reliability of the SA-SH-D.
Method
Following COSMIN guidelines, the study consisted of a cognitive interview study and a questionnaire study. Health care students from nursing, physiotherapy and occupational therapy programmes participated; seven were interviewed following the ‘Think Aloud’ method and 111 responded to two paper-based SA-SH-D administered with a two-week interval.
Results
The SA-SH-D was found to be relevant, comprehensive and comprehensible. Three suboptimal wordings were identified. Floor and ceiling effects were found for six items. Cronbach's α for the total scale was 0.84 indicating good internal consistency. After a two-week interval, the percentage of agreement per item ranged from 88.2% to 100% when allowing for a one-point difference. The mean agreement percentage for the overall scale was 95.2% when allowing for a one-point difference.
Conclusion
The Danish version of Health Care Students Attitudes towards Addressing Sexual Health is a valid and reliable questionnaire for measuring health care professional students’ attitudes towards addressing sexual health in their future professional practice and care.
Uvod: Preporučene smjernice za izvještavanje u sustavnim pregledima i metaanalizama (engl. Preferred Reporting Items for Systematic reviews and Meta-Analyses, PRISMA) prvi su put objavljene 2009. godine kako bi autorima sustavnih pregleda omogućile transparentno izvještavanje o predmetu istraživanja, korištenim metodama i dobivenim rezultatima. Tijekom posljednjeg desetljeća postignut je značajan napredak u metodologiji i terminologiji sustavnih pregleda, što je rezultiralo potrebom za nadopunjavanjem i prilagođavanjem postojećih smjernica. Svrha rada: Predstaviti ažurirane smjernice PRISMA 2020. za izvještavanje u sustavnim pregledima. Metode: Pregledali smo 60 dokumenata sa smjernicama za izvještavanje u sustavnim pregledima s ciljem obuhvaćanja predloženih izmjena smjernica PRISMA iz 2009. godine. Tražili smo povratne informacije o predloženim izmjenama pomoću internetske ankete provedene među 110 urednika časopisa i metodologa sustavnih pregleda. O rezultatima anketa i pregleda raspravljalo se na sastanku na kojem je sudjelovao 21 član. Nakon sastanka, temeljem povratnih informacija koautora i prigodnog uzorka od 15 recenzenata sustavnih pregleda, generirali smo koncept za kontrolni popis PRISMA 2020., sažetak kontrolnog popisa, razradu i objašnjenje te dijagram toka. Rezultati: U ovom radu predstavljamo kontrolni popis za smjernice PRISMA 2020. sastavljene od 27 stavki, odnosno prošireni kontrolni popis s detaljnim preporukama za izvještavanje pojedine stavke, sažetak kontrolnog popisa PRISMA 2020. te revidirane dijagrame toka za izvorne i ažurirane preglede. Kontrolni popis uključuje nove smjernice za izvještavanje koje sadrže ažurirane metode identifikacije, odabira, procjene i sinteze istraživanja. Struktura i prikaz stavaka su izmijenjeni kako bi se olakšala njihova provedba. Smjernice PRISMA 2020. zamjenjuju smjernice iz 2009. godine. Zaključak: Svrha smjernica PRISMA 2020. je olakšati transparentnost, cjelovitost i točnost izvještavanja u sustavnim pregledima. Poboljšano izvještavanje koristit će autorima sustavnih pregleda, autorima smjernica, kreatorima politike, zdravstvenim radnicima, pacijentima i drugim korisnicima. U svrhu postizanja navedenih ciljeva potičemo autore, urednike i recenzente da usvoje predstavljene smjernice.
In the transition towards higher levels of vehicle automation, one of the key concerns with regards to human factors is to avoid mode confusion, when drivers misinterpret the driving mode and therewith misjudge their own tasks and responsibility. To enhance mode awareness, a clear human centered Human Machine Interface (HMI) is essential. The HMI should support the driver tasks of both supervising the driving environment when needed and self-regulating their non-driving related activities (NDRAs). Such support may be provided by either presenting continuous information on automation reliability, from which the driver needs to infer what task is required, or by presenting continuous information on the currently required driving task and allowed NDRA directly. Additionally, it can be valuable to provide continuous information to support anticipation of upcoming changes in the automation mode and its associated reliability or required and allowed driver task(s). Information that could support anticipation includes the available time until a change in mode (i.e. time budget), information on the upcoming mode, and reasons for changing to the upcoming mode. The current work investigates the effects of communicating this potentially valuable information through HMI design. Participants received information from an HMI during simulated drives in a simulated car presented online (using Microsoft Teams) with an experimenter virtually accompanying and guiding each session. The HMI either communicated on automation reliability or on the driver task, and either included information supporting anticipation or did not include such information. Participants were thinking aloud during the simulated drives and reported on their experience and preferences afterwards. Anticipatory information supported understanding about upcoming changes without causing information overload or overreliance. Moreover, anticipatory information and information on automation reliability, and especially a combination of the two, best supported under-standability and usability. Recommendations are provided for future work on facilitating supervision and NDRA self-regulation during automated driving through HMI design.
Critical reading is required for information literacy in the 21st century. However, how students read and process conflicting information regarding controversial issues is still not clear. Using eye-tracking technology, this exploratory study aimed to identify visual behavior patterns that characterized critical reading strategies when students processed conflicting information about socio-scientific issues, and to examine relationships among students' visual behavior, reading strategies, and reading task outcomes. Forty-eight undergraduate and graduate students participated and read conflicting texts about food-science issues. The conflicting texts were presented in the traditional Chinese language. Participants' eye movements were tracked during the entire reading process and their self-reported critical reading strategies, recall, and opinions on the issues were collected right after reading. Pearson's correlation analyses, lag sequential analyses, and content analyses were used to analyze the data. Results showed that participants with higher scores on self-reported critical reading strategies tended to pay more attention to reasoning information in conflicting reports, and to include more judgments in their responses. Further, two significant visual behavior patterns were identified for critical reading: (1) after a pause for thinking or resting, critical readers' visual attention shifted directly back to reasoning information; and (2) critical readers' visual attention shifted between reasoning and data information. These two visual behavior patterns suggested that data inspection is important for critical reasoning about conflicting information. This study provided evidence for the applicability of eye-tracking technology in assessing critical reading strategies and relationships between eye-tracking data and self-reported critical reading strategies. Findings can contribute to the future design of adaptive learning systems and teaching of critical reading strategies.
Objective:
To further refine the wording of screening questions and examine their face validity through cognitive interviews with axial spondyloarthritis (axSpA) and chronic mechanical back pain patients.
Methods:
In-depth, semi-structured cognitive interviews were conducted with 30 patients (10 axSpA; 20 chronic mechanical back pain patients) to assess the face validity and comprehensibility of the screening questions. The interview protocol focused on 12 questions/domains including participants' feedback/thoughts on the duration of suffering from back pain, age at onset of back pain, pace of back pain development, improvement of pain with movement or rest, nocturnal back pain improving upon awakening, pain in other parts of the body, responsiveness of pain to nonsteroidal anti-inflammatory drug (NSAID) use, history of autoimmune conditions, and domains such as sleep, sitting, and stiffness. The Flesch-Kincaid grade level and Flesch reading ease scores were then analyzed for the revised versions of screening questions.
Results:
Participants preferred questions that allowed them to provide more details regarding the frequency of their symptoms. Questions were refined for clarity and eliminated if participants considered them to be irrelevant (e.g., NSAIDs). Two sample screeners were derived from twelve questions each with an overall reading grade of 7.5 and reading ease of 65.7%.
Conclusions:
It is feasible to design a screening tool that is accessible to most (e.g., reading level) and clear to individuals with back pain. An evidence-based approach to demonstrate the validity of the screening tool will be critical for it to be implemented widely into clinical practice. Key Points • Our study developed two sample screeners that are clear to individuals with back pain and accessible to most with an overall Flesch-Kincaid reading grade of 7.5 and Flesch reading ease of 65.7%. • Questions that were considered irrelevant to participants were eliminated such as responsiveness of pain to nonsteroidal anti-inflammatory drug (NSAID). • It is feasible to design a screening tool that is accessible to most (e.g., reading level) and clear to individuals with back pain.
Background
Entrustable Professional Activities (EPAs) assessments measure learners’ competence with an entrustment or supervisory scale. Designed for workplace-based assessment EPA assessments have also been proposed for undergraduate medical education (UME), where assessments frequently occur outside the workplace and may be less intuitive, raising validity concerns. This study explored how assessors make entrustment determinations in UME, with additional specific comparison based on familiarity with prior performance in the context of longitudinal student-assessor relationships.
Methods
A qualitative approach using think-alouds was employed. Assessors assessed two students (familiar and unfamiliar) completing a history and physical examination using a supervisory scale and then thought-aloud after each assessment. We conducted a thematic analysis of assessors’ response processes and compared them based on their familiarity with a student.
Results
Four themes and fifteen subthemes were identified. The most prevalent theme related to “student performance.” The other three themes included “frame of reference,” “assessor uncertainty,” and “the patient.” “Previous student performance” and “affective reactions” were subthemes more likely to inform scoring when faculty were familiar with a student, while unfamiliar faculty were more likely to reference “self” and “lack confidence in their ability to assess.”
Conclusions
Student performance appears to be assessors’ main consideration for all students, providing some validity evidence for the response process in EPA assessments. Several problematic themes could be addressed with faculty development while others appear to be inherent to entrustment and may be more challenging to mitigate. Differences based on assessor familiarity with student merits further research on how trust develops over time.
We developed digital materials for inclusive primary school content lessons that combine factual texts with various elements known to foster reading comprehension (e.g., reading strategies, cooperative learning, glossaries). To investigate the usability and feasibility of the digital learning environment, we conducted two mixed-methods studies in both individual (Study 1) and whole-classroom settings (Study 2). Use of the new digital material was found to be intuitive and motivating. Although teachers and students needed time to familiarize themselves with new functions, they liked working with the material. Students benefited when teachers introduced the material in a clearly structured manner. Teachers emphasized the importance of the cooperative learning methods and appreciated the inclusive characteristics of the approach.
In phase I trials, some biospecimens are used both for research and patient care and some for research only. Some research participants have therapeutic misconception, assuming all biospecimens are for patient care. This study's aim was to test if a simple information chart would improve understanding of nontherapeutic research procedures. A two‐arm study was conducted. Participants in the control group (C) were asked whether biospecimens were for their care, for research only, or for both. The experimental group (E) was asked the same questions but provided with a study‐specific information chart labeling the purpose of each biospecimen. One hundred one patients were interviewed. In both arms, understanding that pretreatment blood draws were for patient care and research was moderate (49% for C and 62% for E). Understanding that posttreatment blood draws were for research only was significantly higher in the experimental arm (16% for C and 44% for E; p = 0.002). Providing a simple information chart may help alleviate this aspect of therapeutic misconception.
Using pre‐ and post‐ intervention data, this formative study explores instructional moves designed to guide students' critical thinking as they engage in lateral reading and digital source evaluation. Working in a dual enrollment composition classroom, researchers identified and addressed three areas of need: considering context when determining source authority or credibility, developing skilled web searching strategies and engaging in complex and multi‐stepped lateral reading moves. Results demonstrated purposeful modeling and direct instruction of related digital skills positioned participants for more nuanced and developed thinking when engaging in digital source evaluation through lateral reading.
Background : Attendance rates for eye clinics are low across low- and middle-income countries (LMICs) and exhibit marked sociodemographic (SD) inequalities. We aimed to quantify the association between a range of SD domains and attendance rates from vision screening in programmes launching in Botswana, Kenya and Nepal.
Methods : We will develop a set of sociodemographic questions and introduce them into routine community-based eye screening programmes in Kenya, Botswana and Nepal, targeting children aged 5-18 years and adults. Our study design is a rolling survey, embedded within the Peek screening programme. The sociodemographic questions will be asked of 10% of all those presenting to be screened, and 100% of those identified with an eye problem. We will also collect data on whether people referred to ophthalmic clinic for treatment or further assessment attended, and we will use logistic regression to report odds ratios for this outcome attendance) for each socioeconomic domain in each country. We hypothesise that attendance rates will be lowest among marginalised sociodemographic groups such as older, less educated, less wealthy women. To identify the most appropriate sociodemographic items we will perform a literature review, and then hold workshops with researchers, academics, programme implementers, and programme designers in each country to tailor the domains and response options to the national context. We will report outcome data at 6 and 12 months, identifying the groups facing the highest barriers to access.
Discussion : This low-risk, embedded, pragmatic, observational data collection will enable eye screening programme managers to accurately identify which sociodemographic groups are facing the highest systematic barriers to accessing care at any point in time. This information will be used to inform the development of service improvements to improve equity.
Designing games is a complicated and time-consuming process, where developing new levels for existing games can take weeks. Procedural content generation offers the potential to shorten this timeframe, however automated design tools are not adopted widely in the game industry. This paper presents an expert evaluation of a human-in-the-loop generative design approach for commercial game maps that incorporates multiple computational agents. The evaluation aims to gauge the extent to which such an approach could support and be accepted by human game designers, and to determine whether the computational agents improve the overall design. To evaluate the approach, eleven game designers utilized the approach to design game levels with the computational agents both active and inactive. Eye tracking, observational and think aloud data was collected to determine whether designers favored levels suggested by the computational agents. This data was triangulated with qualitative data from semi-structured interviews that were used to gather overall opinions of the approach. The eye tracking data indicates that the participating game level designers showed a clear preference for levels suggested by the computational agents, however expert designers in particular appeared to reject the idea that the computational agents are helpful. The perception of computational tools not being useful needs to be addressed if procedural content generation approaches are to fulfill their potential for the game industry.
Molecular tumor boards are an emerging platform for multidisciplinary oncology care specialists to assess treatment options based on the patient’s individual molecular tumor profile. However, they require complex manual preparation, e.g., data retrieval from widespread knowledge bases. We define clinical process models and a software prototype supporting the adoption of virtual molecular tumor boards across multiple clinical sites. Together with real-world experts, we created software prototypes to optimize the individual steps of preparing, conducting, and follow-up after molecular tumor boards. Thus, our Web-based prototype supports oncologists in selecting individual treatment options more effectively.
ResearchGate has not been able to resolve any references for this publication.