No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient information
... A careful assessment of the quality of health information is important for non-professionals seeking accurate and credible knowledge on various health issues. Several tools and guidelines have been developed to achieve this purpose, including the DISCERN tool devised by Charnock et al. to assess the quality of written health information [13], the Centers for Disease Control and Prevention (CDC) Clear Communication Index that helps in developing and evaluating public communication products [14], the Universal Health Literacy Precautions Toolkit, which helps to tailor delivery of care based on variable ranges of health literacy [15], and the Patient Education Materials Assessment Tool, which evaluates the understandability and actionability of printable and audiovisual patient education materials [16]. ...
... We conducted a literature review on the existing instruments for evaluating health information quality to design the intended tool for the assessment of health information generated in AI-based models [13][14][15][16]. This literature review was directed to cover the following aspects: health literacy, information accuracy, clarity, and relevance in health communication. ...
... This literature review was directed to cover the following aspects: health literacy, information accuracy, clarity, and relevance in health communication. The literature search was conducted on PubMed, Medical Literature Analysis and Retrieval System Online (MEDLINE), and Google Scholar databases and concluded on November 1, 2023 [13][14][15][16][17][18][19]. ...
Background
Artificial intelligence (AI)-based conversational models, such as Chat Generative Pre-trained Transformer (ChatGPT), Microsoft Bing, and Google Bard, have emerged as valuable sources of health information for lay individuals. However, the accuracy of the information provided by these AI models remains a significant concern. This pilot study aimed to test a new tool with key themes for inclusion as follows: Completeness of content, Lack of false information in the content, Evidence supporting the content, Appropriateness of the content, and Relevance, referred to as "CLEAR", designed to assess the quality of health information delivered by AI-based models.
Methods
Tool development involved a literature review on health information quality, followed by the initial establishment of the CLEAR tool, which comprised five items that aimed to assess the following: completeness, lack of false information, evidence support, appropriateness, and relevance. Each item was scored on a five-point Likert scale from excellent to poor. Content validity was checked by expert review. Pilot testing involved 32 healthcare professionals using the CLEAR tool to assess content on eight different health topics deliberately designed with varying qualities. The internal consistency was checked with Cronbach's alpha (α). Feedback from the pilot test resulted in language modifications to improve the clarity of the items. The final CLEAR tool was used to assess the quality of health information generated by four distinct AI models on five health topics. The AI models were ChatGPT 3.5, ChatGPT 4, Microsoft Bing, and Google Bard, and the content generated was scored by two independent raters with Cohen's kappa (κ) for inter-rater agreement.
Results
The final five CLEAR items were: (1) Is the content sufficient?; (2) Is the content accurate?; (3) Is the content evidence-based?; (4) Is the content clear, concise, and easy to understand?; and (5) Is the content free from irrelevant information? Pilot testing on the eight health topics revealed acceptable internal consistency with a Cronbach's α range of 0.669-0.981. The use of the final CLEAR tool yielded the following average scores: Microsoft Bing (mean=24.4±0.42), ChatGPT-4 (mean=23.6±0.96), Google Bard (mean=21.2±1.79), and ChatGPT-3.5 (mean=20.6±5.20). The inter-rater agreement revealed the following Cohen κ values: for ChatGPT-3.5 (κ=0.875, P<.001), ChatGPT-4 (κ=0.780, P<.001), Microsoft Bing (κ=0.348, P=.037), and Google Bard (κ=.749, P<.001).
Conclusions
The CLEAR tool is a brief yet helpful tool that can aid in standardizing testing of the quality of health information generated by AI-based models. Future studies are recommended to validate the utility of the CLEAR tool in the quality assessment of AI-generated health-related content using a larger sample across various complex health topics.
... Careful assessment of the quality of health information is important for non-professionals seeking accurate and credible knowledge on health issues. Various tools and guidelines have been developed to achieve this purpose including: the DISCERN tool [13], the CDC Clear Communication Index [14], the Universal Health Literacy Precautions Toolkit [15], and the Patient Education Materials Assessment Tool among other tools [16]. ...
... We conducted a literature review on the existing instruments for evaluating health information quality to design the intended tool for assessment of health information generated in AI-based models [13][14][15][16]. This literature review was directed to cover the following aspects: health literacy, information accuracy, clarity, and relevance in health communication. ...
... This literature review was directed to cover the following aspects: health literacy, information accuracy, clarity, and relevance in health communication. The literature search was conducted on PubMed/MEDLINE and Google Scholar databases and concluded on 1 November 2023 [13][14][15][16][17][18][19]. ...
Background
Artificial intelligence (AI)-based conversational models, such as ChatGPT, Microsoft Bing, and Google Bard, emerged as valuable sources of health information for the lay individuals. However, the accuracy of information provided by these AI models remains a significant concern. This pilot study aimed to test a new tool referred to as “CLEAR”, designed to assess the quality of health information delivered by AI-based models.
Methods
Tool development involved a literature review on health information quality, followed by initial establishment of the CLEAR tool comprising five items that aimed to assess the following: completeness of content in response to the prompt, lack of false information, evidence support, appropriateness, and relevance of the generated content. Each item was scored on a 5-point Likert scale from excellent to poor. Content validity was checked by expert review of the initial items. Pilot testing involved 32 healthcare professionals using the CLEAR tool to assess content on eight different health topics deliberately designed with varying qualities. The internal consistency was checked using the Cronbach α. Feedback through the pilot test resulted in language modifications to improve the clarity of the items. The final CLEAR tool was used to assess health information quality generated through four different AI-based models in five different, yet common health topics. The AI models were ChatGPT 3.5, ChatGPT 4, Bing, and Bard, and the content generated was scored by two independent raters with Cohen κ to assess the inter-rater agreement.
Results
The final five CLEAR items were: (1) Is the content sufficient? (2) Is the content accurate? (3) Is the content evidence-based? (4) Is the content clear, concise, and easy to understand? and (5) Is the content free from irrelevant information? Pilot testing using the eight different health topics revealed an acceptable internal consistency with a Cronbach α range of 0.669–0.981. The use of the final CLEAR tool yielded the following average scores: Bing (mean=24.4±0.42), ChatGPT-4 (mean=23.6±0.96), Bard (mean=21.2±1.79), and ChatGPT-3.5 (mean=20.6±5.20). The inter-rater agreement revealed the following Cohen κ values: for ChatGPT-3.5 (κ=0.875, P<.001), ChatGPT-4 (κ=0.780, P<.001), Bing (κ=0.348, P=.037), and Bard (κ=.749, P<.001).
Conclusions
The CLEAR tool is a brief yet helpful tool that can aid to standardize testing of the quality of health information generated by the AI-based conversational models. Future studies are recommended to validate the utility of the CLEAR tool to assess the quality of the AI-generated health-related content using a larger sample across various complex health topics.
... Careful assessment of the quality of health information is important for non-professionals seeking accurate and credible knowledge on health issues. Various tools and guidelines have been developed to achieve this purpose including: the DISCERN tool [13], the CDC Clear Communication Index [14], the Universal Health Literacy Precautions Toolkit [15], and the Patient Education Materials Assessment Tool among other tools [16]. ...
... We conducted a literature review on the existing instruments for evaluating health information quality to design the intended tool for assessment of health information generated in AI-based models [13][14][15][16]. This literature review was directed to cover the following aspects: health literacy, information accuracy, clarity, and relevance in health communication. ...
... This literature review was directed to cover the following aspects: health literacy, information accuracy, clarity, and relevance in health communication. The literature search was conducted on PubMed/MEDLINE and Google Scholar databases and concluded on 1 November 2023 [13][14][15][16][17][18][19]. ...
Artificial intelligence (AI)-based conversational models, such as ChatGPT, Microsoft Bing, and Google Bard, emerged as valuable sources of health information for the lay individuals. However, the accuracy of information provided by these AI models remains a significant concern. This pilot study aimed to test a new tool referred to as “CLEAR”, designed to assess the quality of health information delivered by AI-based models. Tool development involved a literature review on health information quality, followed by initial establishment of the CLEAR tool comprising five items that aimed to assess the following: completeness of content in response to the prompt, lack of false information, evidence support, appropriateness, and relevance of the generated content. Each item was scored on a 5-point Likert scale from excellent to poor. Content validity was checked by expert review of the initial items. Pilot testing involved 32 healthcare professionals using the CLEAR tool to assess content on eight different health topics deliberately designed with varying qualities. The internal consistency was checked using the Cronbach α. Feedback through the pilot test resulted in language modifications to improve the clarity of the items. The final CLEAR tool was used to assess health information quality generated through four different AI-based models in five different, yet common health topics. The AI models were ChatGPT 3.5, ChatGPT 4, Bing, and Bard, and the content generated was scored by two independent raters with Cohen κ to assess the inter-rater agreement. The final five CLEAR items were: (1) Is the content sufficient? (2) Is the content accurate? (3) Is the content evidence-based? (4) Is the content clear, concise, and easy to understand? and (5) Is the content free from irrelevant information? Pilot testing using the eight different health topics revealed an acceptable internal consistency with a Cronbach α range of 0.669–0.981. The use of the final CLEAR tool yielded the following average scores: Bing (mean=24.4±0.42), ChatGPT-4 (mean=23.6±0.96), Bard (mean=21.2±1.79), and ChatGPT-3.5 (mean=20.6±5.20). The inter-rater agreement revealed the following Cohen κ values: for ChatGPT-3.5 (κ=0.875, P<.001), ChatGPT-4 (κ=0.780, P<.001), Bing (κ=0.348, P=.037), and Bard (κ=.749, P<.001). The CLEAR tool is a brief yet helpful tool that can aid to standardize testing of the quality of health information generated by the AI-based conversational models. Future studies are recommended to validate the utility of the CLEAR tool to assess the quality of the AI-generated health-related content using a larger sample across various complex health topics.
... In order to evaluate overall video quality, a five-point GQS analysis was performed (Table 3). 17 18 The power of the PEMAT is that not only printed materials but also audiovisual materials (PEMAT-A/V) could be evaluated. This scoring system offers mutual and exclusive evaluation criteria for printed and audiovisual materials. ...
... For this reason, it is important to evaluate the understandability and actionability of the examined samples, as well as the quality and content, especially in such studies. In this study, the PEMAT-A/V tool scoring developed by Shoemaker et al. 18 , which was used to evaluate the understandability and actionability of audiovisual materials, was used to evaluate these two parameters 18 . The advantage of this evaluation method is that it gives a general quality level result and gives an idea to the researcher in separate criteria such as understandability and actionability. ...
... For this reason, it is important to evaluate the understandability and actionability of the examined samples, as well as the quality and content, especially in such studies. In this study, the PEMAT-A/V tool scoring developed by Shoemaker et al. 18 , which was used to evaluate the understandability and actionability of audiovisual materials, was used to evaluate these two parameters 18 . The advantage of this evaluation method is that it gives a general quality level result and gives an idea to the researcher in separate criteria such as understandability and actionability. ...
Objectives: This study will assess the quality, understandability and actionability of YouTube TM videos relating to postoperative care, using tooth extraction as a point of focus. Materials and Methods: As keywords, 'postoperative care after tooth extraction' and 'postoperative instructions after tooth extraction' were used. After selection of the videos, a 16-point usefulness index was used in order to evaluate the content of the videos. Modified DICERN and Global Quality Scale (GQS) were used for assessing quality of the videos and in order to evaluate understandability and actionability of the selected videos, Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V) were used. The relationship and correlation between the descriptive data of the videos and the findings of scoring systems and the correlation between scoring systems were evaluated. Results: Totally, 55 videos were selected for evaluation. Duration of the videos was the sole variable affecting the usefulness and quality of the videos (p<0.05). Among evaluated videos 27.3% were very useful and 54.5% were moderately useful. There was a relationship between usefulness and quality-measuring scoring systems, but no relationship between usefulness and PEMAT-A/V scores of the videos (p<0.001; p=0.064 respectively). Conclusion: It could be concluded that videos on YouTube TM about the topic 'postoperative care after tooth extraction' might be useful in an acceptable level, but these selected videos might not be satisfyingly understandable and action-motivating.
... In order to evaluate overall video quality, a five-point GQS analysis was performed (Table 3). 17 18 The power of the PEMAT is that not only printed materials but also audiovisual materials (PEMAT-A/V) could be evaluated. This scoring system offers mutual and exclusive evaluation criteria for printed and audiovisual materials. ...
... For this reason, it is important to evaluate the understandability and actionability of the examined samples, as well as the quality and content, especially in such studies. In this study, the PEMAT-A/V tool scoring developed by Shoemaker et al. 18 , which was used to evaluate the understandability and actionability of audiovisual materials, was used to evaluate these two parameters 18 . The advantage of this evaluation method is that it gives a general quality level result and gives an idea to the researcher in separate criteria such as understandability and actionability. ...
... For this reason, it is important to evaluate the understandability and actionability of the examined samples, as well as the quality and content, especially in such studies. In this study, the PEMAT-A/V tool scoring developed by Shoemaker et al. 18 , which was used to evaluate the understandability and actionability of audiovisual materials, was used to evaluate these two parameters 18 . The advantage of this evaluation method is that it gives a general quality level result and gives an idea to the researcher in separate criteria such as understandability and actionability. ...
... Reviewers were blinded to each other's evaluations. The following video quality assessment tools were used: the Patient Education Materials Assessment Tool for audio-visual content (PEMAT A/V), the DISCERN score and the Misinformation score [19,20]. ...
... The total score was presented as a percentage obtained by the sum of all points, divided by the number of the items judged as agree or disagree. Higher scores detect more understandable and actionable content (a threshold to define a good video quality for PEMAT is a score greater than 70%) [19]. ...
... The Actionability score reflects how viewers could use the information presented. According to Shoemaker et al., videos with PEMAT A/V Understandability and Actionability scores of <70% are considered poorly understandable or poorly actionable [19]. Therefore, only videos related to avanafil, which listed less than 10% of the overall selected videos, resulted in enough understandable, but not enough actionable. ...
Several previous studies on YouTubeTM information on medical topics have already been published. The current study aimed to evaluate the quality information of YouTubeTM videos on Phosphodiesterase 5 inhibitors (PDE5-is). A systematic search on YouTube™ was conducted using 30 keyword combinations. For each keyword’s combination, the first 50 videos were recorded. The quality of videos on YouTube™ was assessed with Patient Education Materials Assessment Tool for audio‐visual content (PEMAT A/V), DISCERN score and a specified created Misinformation tool. According to the selection criteria, 229 YouTube™ videos were suitable for the analyses. Videos were stratified according to the five main identified topics: sildenafil [n = 98; 42.79%] vs tadalafil [n = 50; 21.83%] vs vardenafil [n = 44;19.21%] vs avanafil [n = 17; 7.42%] vs PDE5-is in general [n = 20; 8.73%]. The median overall PEMAT A/V Understandability score and Actionability score were 55% (interquartile range [IQR]: 42–75) and 0% (IQR = 0–67), respectively. Specifically, according to our stratification, YouTube Videos on avanafil reached higher values of both Understandability and Actionability (72.7% and 66.7, respectively) in contrast to other categories. According to DISCERN tool, the total overall median score was 29.5 (IQR = 18–41). According to Misinformation scale, the item 1 (‘sexual stimulation’) harboured an overall median score of 2 (IQR = 1–2); the item 2 (‘side effects’) an overall median score of 2 (IQR = 1–3); the item 3 (‘treatment choices’) an overall median score of 1 (IQR = 1–2); the item 4 (‘contraindications’) an overall median score of 2 (IQR = 1–2). YouTube™ is a fast and open-access source for mass information. The overall quality of the PDE5-is contents provided is sadly unsatisfactory. Nowadays, YouTube™ cannot be recommended as a reliable source of information on PDE5-is.
... While the rate of obtaining information from platforms such as Tiktok, YouTube, etc. has increased, the importance of their quality has also increased (Chen et al., 2022;Ming et al., 2023;Yeung et al., 2022). Patient Education Materials Assessment Tool (PEMAT), a tool designed to assess all of these, was developed by Shoemaker et al. (2014a) to assess "understandability" and "actionability" in both print and audiovisual materials. Understandability relates to the extent to which consumers from different backgrounds can process and explain key messages. ...
... Understandability relates to the extent to which consumers from different backgrounds can process and explain key messages. Actionability refers to the extent to which consumers from different backgrounds and with different levels of health literacy can determine what actions they should take based on the information presented (Shoemaker et al., 2014a). Health information that lacks understandability, reliability, and actionability can lead to serious consequences by affecting patients' DOI: 10.5152/FNJN.2023.22196 ...
... Even if a majority of the elements are related to printable and audiovisual materials, some elements are valid only for one type of material. Consequently, there are two versions of the PEMAT, the "Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P)" composed of 24 items, and the "Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V)" composed of 17 items (Shoemaker et al., 2014a). ...
AIM: The aim of this study was to test the psychometric properties of the Turkish version of the Patient Education Materials Assessment Tool
METHODS: The study was carried out with intern nurses receiving education at a Nursing Faculty in Turkey between July 2019 and June 2020.
248 intern nurses participated in PEMAT-P and 223 in PEMAT-A/V in this study. Five audio-visual and five printed materials were used. The
Patient Education Materials Assessment Tool and the Patient Education Materials Assessment Tool User’s Guide were translated into Turkish
and culturally adapted using standard guidelines. Content and construct validity analyses were performed for the validity of the Turkish version of
the Patient Education Materials Assessment Tool, and internal consistency, invariability, and external consistency analyses were performed for its
reliability.
RESULTS: The content validity index was 1.00 for all the scale items. At the conclusion of the factor analyses, the scale has composed a twodimension construct of understandability and actionability. The Cronbach’s alpha coefficients were observed to be 0.901 of the Turkish version
of the Patient Education Materials Assessment Tool for Printable Materials and 0.897 of the Turkish version of the Patient Education Materials
Assessment Tool for Audiovisual Materials. The Pearson Product-Moment Correlation Coefficient (rxy) was high (Turkish version of the Patient
Education Materials Assessment Tool for Printable Materials rxy=0.815, Turkish version of the Patient Education Materials Assessment Tool for
Audiovisual Materials rxy=0.804). Cohen’s kappa coefficients were obtained as 0.736 of the Turkish version of the Patient Education Materials
Assessment Tool for Printable Materials and 0.781 of the Turkish version of the Patient Education Materials Assessment Tool for Audiovisual
Materials.
CONCLUSION: The Turkish version of the Patient Education Materials Assessment Tool is a valid and reliable measurement tool in the evaluation
of both printed and audiovisual patient education materials
... A Patient Education Materials Assessment Tool (PEMAT-A/V) [39] was used to evaluate audio-visual patient education materials. The PEMAT-A/V tool (English) was 16:116 validated by the Agency for Healthcare Research and Quality [40]. This tool had a moderate agreement per Kappa (average K = 0.57), strong agreement per Gwet's AC1 (average = 0.74), and strong internal consistency (α = 0.71; average item-total correlation = 0.62). ...
... The "Actionability" domain evaluates their ability to take action. The domain of "Understandability" consists of 11 questions after excluding two questions that did not apply to video format, while "Actionability" has 4 questions [40]. ...
... The total score was divided by the total number of items and multiplied by 100% to generate each score for "understandability" and "actionability". A cut-off of 70% is used to determine whether the patient education material in the form of video is sufficiently "understandable" and "actionable" by the end-users [40]. Seventy-nine patients with AR were estimated using a G* power sample size, with an effect size of 0.2 based on Cohen's d [41], a constant proportion of 0.5, a significance level of 0.05, and a power of 0.95. ...
Background
Patient education is identified as one of the core and fundamental management strategies in the management of allergic rhinitis. The Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines developed guidance for the management of allergic respiratory disease, and the guidelines are applicable to the international context. The ARIA guidelines for the pharmacy have specifically encouraged the creation of local pharmacist-led intervention in allergic rhinitis management. This study aims to develop a pharmacist-led educational model using a multi-phase study approach.
Method
In phase one, we conducted a literature review using four databases to extract relevant articles and clinical practice guidelines published between 2017 and 2022. The information was structured into a questionnaire consisting of patient education material (10 domains with 130 items) and pharmacist counseling scopes (15 domains with 43 items), with each item having a rating scale ranging from 1 (lowest) to 9 (highest) level of agreement. Fifty-two panellists, including otorhinolaryngologists and pharmacists, were invited to complete the questionnaire. A consensus agreement was considered when at least 70% of panellists scored 7 to 9 (critically important). A two-round survey was conducted, and descriptive analysis, inter-rater reliability (≥ 0.5–1 indicate moderate to excellent reliability), variation in the relative interquartile (VRIR < 0.3 indicate good stability), and variation in the coefficient of variation (VCV < 40% considered consensus achieved) were performed. In phase two, patient education material was developed into audio-visual format, and in phase three, patients rated its understandability and actionability using a validated Patient Education Materials Assessment Tool.
Results
In the round one Delphi survey, 43 panellists responded, with 171 out of 173 items achieving “consensus agreement” (75.4–100%). In the second survey, 32 out of 43 panellists responded, with most items (171 out of 173 items) stable across rounds and all items had acceptable internal consistency (VCV: − 12.21–15.81). Two items did not achieve “consensus agreement” (64%) but improved in round two (92.9%), however, instability was observed (VRIR: 0.36). These two items were retained in the model due to achieving the minimum level of agreement and internal consistency (VCV = 15.81). Inter-rater reliability was 0.608 and 0.970 in the respective rounds. Patients rated the educational material as understandable (81.8–100%) and actionable (100%).
Conclusion
The validated pharmacist-led education model, with its educational materials tested on end-users, provides structured patient education and pharmaceutical care in assisting patients with allergic rhinitis. The educational material allows the delivery of standardized information by the healthcare providers to the patients. Further research on the effectiveness of this model in improving patients’ symptom control and quality of life is warranted.
... The PEMAT assesses resources on 2 domains: understandability (PEMAT-P includes 17 items; PEMAT-AV includes 13 items) and actionability (PEMAT-P includes 7 items; PEMAT-AV includes 4 items). 16 We scored the items as 0 (disagree), 1 (agree) or NA (not applicable) according to the PEMAT User's Guide. 16,17 PEMAT scores were calculated as percentages of total possible points (excluding not applicable). ...
... 16 We scored the items as 0 (disagree), 1 (agree) or NA (not applicable) according to the PEMAT User's Guide. 16,17 PEMAT scores were calculated as percentages of total possible points (excluding not applicable). 16 According to PEMAT, materials are considered understandable when consumers of diverse backgrounds and varying levels of health literacy can process and explain the key messages. ...
... 16,17 PEMAT scores were calculated as percentages of total possible points (excluding not applicable). 16 According to PEMAT, materials are considered understandable when consumers of diverse backgrounds and varying levels of health literacy can process and explain the key messages. Resources were considered actionable when consumers of diverse backgrounds and varying levels of health literacy can identify what actions are possible to take based on the information provided. ...
Background:
Medical assistance in dying (MAiD) was legalized in Canada in 2016, with legislation updated in 2021. It is unclear whether resources are available to help patients make this difficult decision; therefore, we sought to identify and quality appraise Canadian MAiD resources for supporting patients making this decision.
Methods:
We conducted an environmental scan by searching Canadian websites for online MAiD resources that were published after the 2016 MAiD legislation, patient targeted, publicly accessible and able to inform decisions about MAiD in Canada. We excluded resources that targeted health care professionals or policy-makers, service protocols and personal narratives. Two authors appraised resources using the International Patient Decision Aids Standards (IPDAS) criteria and the Patient Education Materials Assessment Tool (PEMAT) for health literacy. Descriptive analysis was conducted. We defined resources as patient decision aids if 7 IPDAS defining criteria were met, and we rated resources as adequate for understandability or actionability if the PEMAT score was 70% or greater.
Results:
We identified 80 MAiD resources. As of March 2023, 62 resources (90%) provided eligibility according to the 2021 legislation and 11 did not discuss any eligibility criteria. The median IPDAS score was 3 out of 7; 52% discussed alternative options and none provided benefits or harms. Of 80 resources, 59% were adequate for understandability and 29% were adequate for actionability.
Interpretation:
Although many resources on MAiD were updated with 2021 legislation, few were adequate to support patients with lower health literacy. There is a need to determine whether a patient decision aid would be appropriate for people in Canada considering MAiD.
... The Patient Education Materials Assessment Tool (PEMAT) systematically examines how the required action points are presented. The PEMAT was developed by experts under the direction of the Agency for Healthcare Research and Quality (AHRQ) and demonstrated content validity, internal consistency, and reliability [19]. A content analysis of online materials on CKD lifestyle modification using PEMAT reported that the "actionability" of the materials and the method of presenting visual aids need to be improved [20]. ...
... The understandability and actionability of the material were rated using the Japanese version of the Patient Education Material Evaluation Tool for Printed Materials (PEMAT-P). PEMAT is intended for use by healthcare professionals, health librarians, and other professionals who provide health and medical information to patients and the general public [19,26,27]. As with the original PEMAT [19], the Japanese version of the PEMAT-P has been tested for reliability and validity [28]. ...
... PEMAT is intended for use by healthcare professionals, health librarians, and other professionals who provide health and medical information to patients and the general public [19,26,27]. As with the original PEMAT [19], the Japanese version of the PEMAT-P has been tested for reliability and validity [28]. In particular, it has proven predictive validity, which means that the assessment results are predictive of understandability and actionability for the audience. ...
Background
Previous studies have not fully determined whether online education materials on chronic kidney disease (CKD) for Japanese patients are easy to understand and help change their behavior. Therefore, this study quantitatively assessed the understandability and actionability of online CKD education materials.
Methods
In September 2021, we searched Google and Yahoo Japan using the keywords “kidney,” “kidney disease,” “CKD,” “chronic kidney disease,” and “renal failure” to identify 538 webpages. We used the Japanese version of the Patient Education Materials Assessment Tool (PEMAT), ranging from 0 to 100%, to evaluate the understandability and actionability of webpages. We set the cutoff point to 70%.
Results
Of the 186 materials included, the overall understandability and actionability were 61.5% (± 16.3%) and 38.7% (± 30.6%), respectively. The materials were highly technical in their terminology and lacked clear and concise charts and illustrations to encourage action. Compared to lifestyle modification materials on CKD overview, symptoms/signs, examination, and treatment scored significantly lower on the PEMAT. In addition, the materials produced by medical institutions and academic organizations scored significantly lower than those produced by for-profit companies.
Conclusion
Medical institutions and academic organizations are encouraged to use plain language and to attach explanations of medical terms when preparing materials for patients. They are also expected to improve visual aids to promote healthy behaviors.
... The Patient Education and Materials Assessment Tool (PEMAT) is a reliable and validated instrument designed by Shoemaker et al that evaluates the understandability and actionability of online patient health materials [16]. It assesses factors such as material content, word choice, style, use of numbers, and presentation factors of organization, layout, design of web pages, and inclusion of visual aids. ...
... It assesses factors such as material content, word choice, style, use of numbers, and presentation factors of organization, layout, design of web pages, and inclusion of visual aids. Understandability scores are calculated based on 19 items, determining if individuals with varying health literacy levels can comprehend the central message [16,17]. Actionability, evaluated through seven items, measures the audience's ability to apply the information learned. ...
... Each domain is assigned a percentage score by dividing the sum of awarded points by the total number of applicable points. Higher scores indicate greater understandability and actionability of materials [16]. This tool has been validated by the Agency for Healthcare Research and Quality specifically for assessing the quality of patient education materials [17]. ...
IntroductionBreast reduction surgery aims to alleviate physical discomfort and improve the quality of life for individuals with macromastia. Insurance coverage plays a crucial role in making this surgery accessible, but navigating the complex approval process can be challenging. Online resources have become a primary information source, but limited research exists on the adequacy of online materials, particularly for Spanish-speaking patients. This study evaluates the readability, actionability, and understandability of online educational materials on breast reduction insurance coverage for Spanish- and English-speaking patients.Methods
We conducted an online search using the phrase "breast reduction insurance" and selected the first eight institutional or organizational websites that provided information on breast reduction insurance in both English and Spanish. We evaluated online materials using the Patient Education and Materials Assessment Tool (PEMAT), Cultural Sensitivity Assessment Tool (CSAT), and Simplified Measure of Gobbledygook, Spanish (SOL). These tools were used to assess factors such as understandability and actionability, cultural sensitivity, and readability of the materials.ResultsBoth English and Spanish materials scored high in understandability and actionability, with similar average scores between the languages. Cultural sensitivity scores indicated acceptable materials. However, Spanish materials had a higher reading grade level and more hard words compared to English materials.Conclusion
There is a need for accessible and understandable online resources on breast reduction insurance coverage, particularly for Spanish-speaking patients. While the assessed websites generally provided comprehensible information, improvements can be made to enhance visual aids and simplify language. These improvements can better educate patients, improve outcomes, and reduce healthcare costs.Level of Evidence VThis journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266.
... The material's sections have informative headers 10 The material presents information in a logical sequence 11 The material provides a summary Topic: Layout and design 12 The material uses visual cues (e.g., arrows, boxes, bullets, bold, larger font, highlighting) to draw attention to key points 13 Text on the screen is easy to read 14 ...
... PEMAT-A/V is a reliable and valid instrument to evaluate the understandability and actionability of audiovisual patient education materials 11 . The strong internal-consistency and inter-rater reliability of this tool provide a robust framework for such evaluations. ...
... Furthermore, the tool's construct validity has been bolstered through consumer testing. So, the PEMAT tool could help lay and health professionals select proper education materials that reduce health literacy demands 11 . When both the understandability score and the actionability score on the PEMAT-A/V are below 70 points, the videos might be deemed to be of low quality. ...
More people use the internet for medical information, especially YouTube. Nevertheless, no study has been conducted to analyze the quality of YouTube videos about tinnitus in Korea. This study aims to review the contents and quality of YouTube videos on tinnitus. The top 100 Korean YouTube videos on tinnitus were reviewed by a tinnitus expert. This study assessed video details: title, creator, length, and popularity indicators—subscribers, views, and likes. The contents of the video clips were analyzed to determine the relevance, understandability, actionability, and quality of information. Out of 100 tinnitus videos, 27 were created by otolaryngologists, 25 by traditional Korean medicine doctors, 25 by other medical professionals, and 3 by lay persons. Sensorineural tinnitus was frequently dealt, and hearing loss, stress, and noise were introduced as main causes of tinnitus. Otolaryngologists' videos covered verified treatments, but others suggested unproven therapies including herbal medicine or acupressure. Otolaryngologists' videos showed significantly higher understandability and quality of information compared to others (p < 0.001). This study found that tinnitus YouTube videos frequently present low-quality and incorrect material, which could have an adverse effect on patients. Results highlight the need for tinnitus specialists to provide accurate information.
... Currently, writing in plain language requires a health information provider to manually implement advice from health literacy guidelines and checklists. [9][10][11][12] This is a process that demands considerable expertise and time. Though there are tools for objectively assessing the health literacy of health information and automating text-simplification, [13][14][15] revisions are still largely carried out by humans. ...
... 16 The current study also demonstrated that ChatGPT can support implementation of health literacy guidelines for written health information. [9][10][11][12] Although it is not a complete solution, ChatGPT's strength lies in the speed at which it can redraft plain language content for further review, rather than its ability to generate a 'final' publicfacing resource. ...
Background
Most health information does not meet the health literacy needs of our communities. Writing health information in plain language is time-consuming but the release of tools like ChatGPT may make it easier to produce reliable plain language health information.
Objective
To investigate the capacity for ChatGPT to produce plain language versions of health texts.
Design
Observational study of 26 health texts from reputable websites.
Methods
ChatGPT was prompted to ‘rewrite the text for people with low literacy’. Researchers captured three revised versions of each original text.
Main Measures
Objective health literacy assessment, including Simple Measure of Gobbledygook (SMOG), proportion of the text that contains complex language (%), number of instances of passive voice and subjective ratings of key messages retained (%).
Key Results
On average, original texts were written at grade 12.8 (SD = 2.2) and revised to grade 11.0 (SD = 1.2), p < 0.001. Original texts were on average 22.8% complex (SD = 7.5%) compared to 14.4% (SD = 5.6%) in revised texts, p < 0.001. Original texts had on average 4.7 instances (SD = 3.2) of passive text compared to 1.7 (SD = 1.2) in revised texts, p < 0.001. On average 80% of key messages were retained (SD = 15.0). The more complex original texts showed more improvements than less complex original texts. For example, when original texts were ≥ grade 13, revised versions improved by an average 3.3 grades (SD = 2.2), p < 0.001. Simpler original texts (< grade 11) improved by an average 0.5 grades (SD = 1.4), p < 0.001.
Conclusions
This study used multiple objective assessments of health literacy to demonstrate that ChatGPT can simplify health information while retaining most key messages. However, the revised texts typically did not meet health literacy targets for grade reading score, and improvements were marginal for texts that were already relatively simple.
... We assessed the quality of each educational TikTok video using the PEMAT-A/V scale. This is a validated tool that was developed to identify the understandability and actionability of audiovisual patient educational content [11,12]. The scale is comprised of 13 questions for understandability and four questions for actionability. ...
... Each of the 17 items in the instrument is scored as 0 (disagree) or one (agree), with eight items having an additional option of Not Applicable (NA). The percentage score for each subscale ranges from 0 to 100% and is calculated by dividing the total summed points by the total possible points [11,12]. A low percentage score indicates that the content has deficiencies in terms of understandability and actionability, suggesting that materials may be difficult for patients to comprehend and/or apply to their health-related decision-making, respectively [12]. ...
With the continuous rise of social media usage, more patients are looking online for health-related information. TikTok is one of the fastest-growing video-based social media platforms, but the quality of its ophthalmologic content, at a comprehensive level, has not been previously analyzed. We aim to explore and characterize popular ophthalmology content on TikTok, including the presence of misinformation.
Between April 16 and May 22, 2023, 37 different ophthalmology-relevant hashtags were queried on TikTok, and the top 20 most-liked videos per hashtag were analyzed. The quality of educational videos was graded on understandability and actionability using the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V). Trends in creator identity, content type, engagement metrics, misinformation presence, and TikTok verification status were also assessed.
The 37 ophthalmology-related hashtags yielded 723 videos comprising 3.806 billion views. A minority of videos were created by ophthalmologists (16.9%) and eyecare providers (35.1%), while the majority were created by non-healthcare providers (55.0%). The most common types of videos identified were primarily related to personal experiences (35.8%) and education (38.0%). Amongst educational videos, mean PEMAT-A/V understandability and actionability scores were 88.1% and 50.6%, respectively. Misinformation was found in 5.4% of all videos, comprising 4.8% of all likes, 4.7% of all comments, and 11.7% of all bookmarks. Its presence was significantly correlated with content created by non-healthcare providers (p < 0.001) and received a disproportionately higher percentage of bookmarks.
This was the largest, most comprehensive evaluation of ophthalmologic content on TikTok. A significant portion of popular ophthalmologic content on TikTok is created by non-eyecare providers and contains misinformation. Our findings confirm the need for ophthalmologists to create more engaging, actionable, and factual educational content to combat misinformation adoption.
... And (3), what is the overall rating of the publication? Furthermore, websites were given one of the five grades: excellent (63-75), good (51-62), fair (39-50), poor (27)(28)(29)(30)(31)(32)(33)(34)(35)(36)(37)(38), and very poor (15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26) based on overall scores [14]. ...
... The questions are split into an understandability section, designed to assess if consumers of various backgrounds can understand the key concepts, and an actionability section, designed to assess how much consumers of various backgrounds are able to do with the information and tools provided [18]. Previous studies have used a nonempirical cutoff of 70%, where scores below would be considered poorly actionable and poorly understandable [19,20]. ...
Introduction
Brain arteriovenous malformations (AVMs) are vascular deformities created by improper connections between arteries and veins, most commonly in the brain and spinal cord. The management is complex and patient-dependent; further understanding of patient education activities is imperative. Internet access has become more ubiquitous, allowing patients to utilize a large database of medical information online. Using Google Trends (GT) (Google LLC, Mountain View, CA, USA), one can see the public interest in a particular topic over time. Further, when presented with numerous search results, patients may not be able to identify the highest-yielding resources, making objective measures of information quality and readability imperative.
Methods
A GT analysis was conducted for “hereditary hemorrhagic telangiectasia,” “cerebral aneurysm,” and “arteriovenous malformation”. These relative search volumes (RSV) were compared with the 2017 to 2019 annual USA AVM diagnosis quantity for correlation. These RSVs were also compared with the 2017 to 2019 annual USA deaths due to cerebral hemorrhagic conditions. One search was conducted for “brain arteriovenous malformation”. Since most users looking for health information online use only the first page of sources, the quality and readability analyses were limited to the first page of results on Google search. Five quality tools and six readability formulas were used.
Results
Pearson’s correlation coefficients showed positive correlations between USA AVM RSVs and annual AVM deaths per capita from 2017 to 2019 (R²=0.932). The AVM annual diagnosis quantity and AVM RSVs showed a strong positive correlation as well (R²=0.998). Hereditary hemorrhagic telangiectasia and cerebral aneurysms had strong positive correlations between their RSVs and their corresponding annual diagnoses in the 2017 to 2019 time period (R²=0.982, R²=0.709). One-way ANOVA, for USA’s 2004 to 2021 AVM RSVs and 2004 to 2019 deaths per capita, displayed no month-specific statistically significant repeating pattern (all p>0.483). The DISCERN tool had four websites that qualified as “poor” and five as “good.” The average score for the tool was “good.” The Journal of the American Medical Association (JAMA) benchmark scores were very low on average, as four websites achieved zero points. There was a wide variance in the currency, relevance, authority, accuracy, and purpose (CRAAP) scores, indicating an inconsistent level of webpage reliability across results. The patient education materials assessment tool (PEMAT) understandability (86.6%) showed much higher scores than the PEMAT actionability (54.6%). No readability score averaged at or below the American Medical Association (AMA)-recommended sixth-grade reading level.
Conclusion
These GT correlations may be due to patients and families with new diagnoses researching those same conditions online. The seasonality results reflect that no prior research has detected seasonality for AVM diagnosis or presentation. The quality study showed a wide variance in website ethics, treatment information quality, website/author qualifications, and actionable next steps regarding AVMs. Overall, this study showed that patients are routinely attempting to access information regarding these intracranial conditions, but the information available, specifically regarding AVMs, is not routinely reliable and the reading level required to understand them is too high.
... Several sets of guidelines have been developed to facilitate health communication and to evaluate materials' adherence to these guidelines for accessibility, accuracy, and utility (e.g., SMOG Grading: McLaughlin, 1969; PMOSE/ IKIRSCH Document Readability Formula: Mosenthal & Kirsch, 1998; Assessment for Suitability of Materials [ASM]: Doak et al., 1996;Patient Education Materials Assessment Tool [PEMAT]: Shoemaker et al., 2014; the Health Literacy INDEX: Kaphingst et al., 2012). Researchers have found that guidelines for and tools evaluating online health information each have their own strengths and limitations, and an industry standard has yet to be identified (Beaunoyer et al., 2017;Grene et al., 2017;Stoll et al., 2022). ...
... The Centers for Disease Control and Prevention's healthspecific tool, the Clear Communication Index (Centers for Disease Control & Prevention, 2019), and many other tools that have been developed to evaluate health information, require further evaluation for effectiveness across contexts, chief complaints, and cultures. These health-specific tools incorporate not only assessment of the physical presentation of materials but also include requirements for specific health-related components (e.g., risks and benefits of the behavior; Baur & Prue, 2014;Centers for Disease Control & Prevention, 2019;Doak et al., 1996;Kaphingst et al., 2012;McLaughlin, 1969;Mosenthal & Kirsch, 1998;Shoemaker et al., 2014). It follows, then, that adherence to the Guidelines be the first step towards improving accessibility, with the next step being use of the CCI or similar framework to improve the utility of the information. ...
Research indicates that mindfulness and meditation are effective and inexpensive ways of reducing symptoms of anxiety and depression. People first search for information on the internet. However, the public’s access to clear health information online is occluded by variability in accessibility of this information. This study examined the accessibility of mediation and mindfulness information online; both keywords were investigated as they are often used interchangeably. The final sample (N = 136) included 76 websites from the search term “mindfulness” and 60 from the keyword “meditation.” Sites were evaluated using a codebook based on the Federal Plain Language Guidelines, a policy developed to provide advice on clear communication in writing government documents for the public. Evaluation occurred in three major domains: web page format, clear and concise language, and multimedia. Out of 14 possible categories, the mean adherence for websites obtained with the keyword “mindfulness” was 9.09 (SD = 1.71) and “meditation” was 9.23 (SD = 1.65), yielding a combined overall adherence mean of 9.15 (SD = 1.67). Overall, results indicate that about two-thirds of the Federal Plain Language Guidelines were adhered to in the sample. Future directions include evaluating the accuracy of the information provided, exploring differential accessibility needs of written materials based on format, and assessing the interaction between the written materials and human factors. Policy implications include the clarification of the ADA to include online public spaces, particularly those with health-related information.
... The aim of this study was to present our online video production activities and assess the quality, content, and readability of current health information about hereditary breast cancer on YouTube. Using the following search terms, "hereditary breast cancer (kanji and hiragana notation)" and "HBOC," we evaluated the first 100 YouTube videos and conducted a systematic evaluation of the quality of breast cancer information available on YouTube using validated assessment instruments, including the Patient Education Materials Assessment Tool (PEMAT) [19] and the DIS-CERN quality criteria for consumer health information [20]. Furthermore, we compared the quality of BC Tube content with other online videos. ...
Background
The internet, especially YouTube, has become a prominent source of health information. However, the quality and accuracy of medical content on YouTube vary, posing concerns about misinformation. This study focuses on providing reliable information about hereditary breast cancer on YouTube, given its importance for decision-making among patients and families. The study examines the quality and accessibility of such content in Japanese, where limited research has been conducted.
Methods
A nonprofit organization called BC Tube was established in May 2020 to create informative videos about breast cancer. The study analyzed 85 YouTube videos selected using the Japanese keywords “hereditary breast cancer” and “HBOC”, categorized into six groups based on the source of upload: BC Tube, hospitals/governments, individual physicians, public-interest organizations/companies, breast cancer survivors, and others. The videos were evaluated based on various factors, including content length, view counts, likes, comments, and the presence of advertisements. The content was evaluated using the PEMAT and DISCERN quality criteria.
Results
BC Tube created high-quality videos with high scores on PEMAT understandability, significantly outperforming other sources. Videos from public-interest organizations/companies received the most views and likes, despite their lower quality. Videos from medical institutions and governments were of superior quality but attracted less attention.
Conclusions
Our study emphasizes the importance of promoting accessible, easy-to-understand, and widely recognized medical information online. The popularity of videos does not always correspond to their quality, emphasizing the importance of quality evaluation. BC Tube provides a peer-reviewed platform to disseminate high-quality health information. We need to develop high-quality online health information and encourage the promotion of evidence-based information on YouTube.
... Na literatura (15) , encontram-se inúmeras ferramentas válidas e confiáveis que contribuem para a qualidade das pesquisas, fornecendo um alto nível de rigor científico para o desenvolvimento tecnológico. Como exemplo disso, a literatura internacional aponta importantes estudos para subsidiar a produção tecnológica, tal como a Patient Education Materials Assessment Tool (PEMAT) (16) , utilizada como instrumento sistemático para avaliar e comparar a compreensibilidade e a capacidade de ação de materiais voltados ao processo educativo de pacientes em diferentes contextos. Outro instrumento é o Suitability Assessment of Materials (SAM) (17) , ferramenta já validada para o português (18) , que representa um método sistemático para avaliar objetivamente a adequação de materiais de informação em saúde para pacientes. ...
Objective
to elaborate and validate the Instrumento para Avaliação de Modelos metodológicos voltados ao Desenvolvimento de Tecnologias.
Methods
a methodological study, developed in three stages: instrument structuring through documentary research and researchers’ expertise; instrument validity with a panel of 11 expert judge nurses; and instrument final composition.
Results
the instrument, after validity by experts, consisted of 30 items, divided into the content (26 items) and appearance (four items) domains. In the initial instrument validity process, 11 items were modified and seven were withdrawn, as they had a percentage of agreement below 0.80. The instrument showed excellent internal consistency, with values greater than 0.90 in its psychometric criteria.
Conclusion
the instrument produced and validated aimed at assessing research methodological models for technological development showed good reliability, and may contribute to the methodological rigor of technological development research in nursing.
Descriptors:
Psychometrics; Surveys and Questionnaires; Nursing Methodology Research; Technological Development; Nursing
... In the literature (15) , there are numerous valid and reliable tools that contribute to the quality of research, providing a high level of scientific rigor for technological development. As for an instance, the international literature points to important studies to support technological production, such as the Patient Education Materials Assessment Tool (PEMAT) (16) , used as a systematic instrument to assess and compare the comprehensibility and actionability of materials aimed at patients' educational process in different contexts. Another instrument is the Suitability Assessment of Materials (SAM) (17) , a tool already validated for Portuguese (18) , which represents a systematic method to objectively assess the suitability of health information materials for patients. ...
Objective
to elaborate and validate the Instrumento para Avaliação de Modelos metodológicos voltados ao Desenvolvimento de Tecnologias.
Methods
a methodological study, developed in three stages: instrument structuring through documentary research and researchers’ expertise; instrument validity with a panel of 11 expert judge nurses; and instrument final composition.
Results
the instrument, after validity by experts, consisted of 30 items, divided into the content (26 items) and appearance (four items) domains. In the initial instrument validity process, 11 items were modified and seven were withdrawn, as they had a percentage of agreement below 0.80. The instrument showed excellent internal consistency, with values greater than 0.90 in its psychometric criteria.
Conclusion
the instrument produced and validated aimed at assessing research methodological models for technological development showed good reliability, and may contribute to the methodological rigor of technological development research in nursing.
Descriptors:
Psychometrics; Surveys and Questionnaires; Nursing Methodology Research; Technological Development; Nursing
... The online survey review process included items based on the Patient Education Materials Assessment Tool (PEMAT). 35 The PEMAT was designed to guide the development and selection of patient education materials in the constructs of patients understandability (how easy is it for the participant to understand the material) and actionability (how easy is it for the participant to act on the information). Items for printable materials including questions related to the educational materials content, word choice, organisation and design. ...
Background
Complex regional pain syndrome (CRPS) can be a debilitating pain condition with enduring physical, psychological and social impacts. CRPS is often poorly understood by healthcare professionals and management needs to be tailored to each individual’s presentation. People with lived experience express difficulty in accessing reliable and meaningful information about the condition. This study aimed to co-create a trustworthy infographic to share information about the lived experience of CRPS.
Methods
We adopted a seven-phase, iterative, participatory methodology to co-create the infographic. Potential infographic content was obtained from qualitative work investigating the lived experience of CRPS. Online consumer engagement (people with doctor diagnosed CRPS/their family, n=20) was used to prioritise content to be included in the infographic and then potential designs were sourced. The research team narrowed the selections down to two designs which were presented to consumers online for final selection ( n=25) and refinement ( n=34).
Results
An infographic for understanding the lived experience of CRPS was completed using participatory design, providing a resource aligned to the needs of people with this condition. Using the Patient Education Materials Assessment Tool, the final infographic rated highly for understandability (92%) and participants indicated significant willingness to share this infographic with others (93%).
Conclusion
A process of participatory design was an effective and efficient process for translation of evidence gathered from qualitative research into a trustworthy resource for people with CRPS and their support people.
... Questions came from Shoemaker et al's 2014 Patient Education Materials Assessment Tool (PEMAT) model. 34 All questions were on a 2-point agree/disagree scale and covered content, word choice and style, use of numbers, organization, layout and design, and use of visual aids. ...
Background
Quality training is an oft-cited barrier to effective implementation and ongoing delivery of high-quality evidence-based practice (EBP) across fields. This is especially true in the addiction field, but there is little cited evidence for optimal methods to improve EBP in inpatient addiction facilities with minimal resources.
Objective
The current paper focuses on evaluating the state of our facility’s group CBT manual and clinical training on the manual in a “realistic” (ie, non-RCT, non-grant-funded) inpatient addiction treatment setting.
Methods
Five full-time clinicians volunteered to take part in the study (woman = 60%; Mage = 36.20 years). The study involved a mix of semi-structured interviews and surveys designed to measure seven outcomes (barriers, feasibility, useability, appropriateness, acceptability, burden, trialability).
Results
Three themes emerged from the data that impacted the group CBT manual: training, timing, and functionality. Addressing these themes allowed for a new, optimal manual and training procedure to be put into place.
Conclusion
The current study highlights that under-resourced inpatient addiction facilities can still methodically utilize implementation approaches to study their EBP, namely CBT. Such an approach will ensure that the highest quality care is being delivered to patients and actively addresses known training barriers that prevent proper EBP delivery.
... Qualitative data were analyzed for relevant themes, summarized, and used to guide the iterative processes of co-designing and testing CKD-PEM prototypes and inform materials dissemination [39]. Quantitative data included the overall score of the Patient Education Material Assessment Tool (PEMAT-P) [40] used to evaluate the final CKD-PEM drafts and tracking records of patient CKD screening since the initiation of the study. Recommendations from the CAC were analyzed and implemented throughout the study. ...
Ensuring equitable chronic kidney disease (CKD) education for Latine patients with low health literacy and low English proficiency stands as a critical challenge, and the “Caridad Awareness and Education” (CARE) initiative represents our ongoing effort to address this imperative issue. In collaboration with twenty-three patients living with CKD, diabetes and/or hypertension and twelve trained Community Health Workers (CHWs) from diverse Latine subgroups, we conducted a research initiative funded by the National Kidney Foundation. Our primary objective was to co-design and test culturally tailored patient education materials (PEMs) for underserved Latine adults at risk for or diagnosed with CKD. We effectively integrated Community-Engaged Research (CEnR) principles with a Human-Centered Design (HCD) approach to create a range of CKD-PEM prototypes in Spanish. Patient preferences for printed educational materials were clear. They favored printed materials that incorporated visual content with concise text over digital, email, texts, or online resources and personalized phone outreach and the involvement of CHWs. Additionally, patients identified their unwavering commitment to their families as a forceful motivator for caring for their kidney health. Currently, a culturally and linguistically tailored CKD flipchart for one-on-one education, led by CHWs, is undergoing a pilot testing phase involving a sample of one hundred Latine patients at risk for or diagnosed with CKD. This innovative approach signifies a commitment to amplifying the insights and expertise of the Latine community afflicted by kidney health disparities, effectively embracing a CEnR to forge meaningful and impactful CKD-PEMs.
... The PEMAT-P understandability and actionability scores are calculated by adding all the points, dividing by the total possible points (excluding not applicable items), and multiplying by 100 to obtain a percentage. The threshold for both subdomains was set at 70% [9]. ...
Objective
In Japan, educational materials on the home care of coronavirus disease 2019 (COVID-19) were developed owing to limited access to medical care during the pandemic. This study quantitatively evaluated the understandability, actionability, natural flow, and readability of 87 materials published by local governments in Japan for patients with COVID-19. Their understandability and actionability were rated using the Japanese version of the Patient Education Material Evaluation Tool for Printed Materials (PEMAT-P). Natural flow and readability were rated using Global Quality Score (GQS) and jReadability, respectively.
Results
Of the 87 materials, 55 (62.1%) were understandable and 33 (37.9%) were actionable according to the PEMAT-P. Regarding understandability, the materials used medical terms without providing definitions and lacked summaries. Regarding actionability, the materials did not demonstrate explicit steps or utilize visual aids to help the readers take action. The mean (SD) of GQS was 3.44 (0.98), indicating a moderate level of naturalness and comprehensiveness of the materials. The mean (SD) score for readability was 2.4 (0.6), indicating a “lower advanced” level. However, challenges regarding the materials’ plain language remained, such as defining medical terms, summarizing the content for understandability, and using charts and tables that encourage patients to act.
... Understandability and actionability were scored using the Patient Education Materials Assessment Tool (PEMAT) for audio-visual (A/V) materials [9,10]. Actionability refers to whether the material describes an action the viewer can take, and whether it describes and explains steps toward taking that action. ...
Background
Social media platforms are a significant source of information for young women with breast cancer (YWBC) undergoing post-mastectomy breast reconstruction (PMBR). The emphasis on visual storytelling and building community in online spaces renders YouTube to be a common platform for discussions of BR. This study aims to assess the quality of YouTube videos about PMBR, identify themes related to the PMBR experience, and quantify suggestions of BR as an option PM.
Methods
YouTube was searched incognito with terms “young women breast cancer reconstruction” in 08/2022, in order from most to least viewed. Title, upload date, length, poster identity, number of likes and comments were collected. The Patient Education Materials Assessment Tool for audiovisual materials (PEMAT A/V) was used to evaluate understandability and actionability. DISCERN assessed quality and reliability. Presence of sponsorship, intended audience, patient and healthcare professional narratives, and suggestions of PMBR were collected. Reviewers noted whether PMBR was shown and how. Themes were collected inductively and deductively for thematic analysis.
Results
193 videos were identified. Mean video length was 14.6 minutes (SD 20.0 min). 87.1% included sponsorships. 95.9% of videos were posted by an organization. 60.6% were information-based; 45.6% experience-based. Mean PEMAT scores for understandability and actionability were 71.3% (SD 13.4) and 35.7% (SD 41.8), respectively. Mean DISCERN was 2.6/5 (SD 1.2). Patient narrative was present in 52.6% and healthcare professionals’ in 68.4%. PMBR was visually presented 22.8% of the time. 13.5% of videos explicitly recommended PMBR. 2.6% explicitly discouraged it. Patients (77.7%) represented the majority of the intended audience. Most common deductively identified themes included treatment (87.1%), family relationship (17.1%), motherhood ( 15.5%), fertility (11.9%). Inductively identified subthemes included differentiating between various options for PMBR surgery, BRCA genetic testing, psychosocial effects of breast cancer and PMBR, and recovery from surgery.
Conclusions
YouTube is a platform wherein various PMBR options are widely discussed and explained. PMBR videos are highly understandable but have moderate levels of actionability, quality, and reliability. Videos are highly sponsored, demonstrating significant institutional bias. Themes are overwhelmingly treatment and surgery-based. Personal themes were present but not dominant.
... This is due to the basis of the formula for scores like the Flesch-Kincaid grade levels, which take into account the number of syllables per word. In an effort to cater to those with lower literacy levels, the COPE-SCD website utilised different tools to break up text and make it more accessible to the reader, which has been recommended by patient education checklists [31]. This included graphics and design features to draw attention to key points and reduce the amount of text on the page. ...
Objective: To develop content for the website component of COPE-SCD: an online community supporting families after sudden cardiac death (SCD) and assess its acceptability.
Methods: Based on topics from codesign focus groups, website content was drafted, and readability of language assessed. SCD family members were recruited from the Genetic Heart Disease Clinic, Royal Prince Alfred Hospital, Sydney. Demographic and psychological measures were collected at baseline. Think aloud interviews were conducted to assess the website with questions mapped to theoretical framework of acceptability.
Results: Fourteen webpages were created under six menu options. Topics range from information on SCD, genetic testing, grief and loss and family stories. Average Flesch-Kincaid school grade reading score was 8.75. Six individuals participated in a think aloud analysis and interview. Fifty percent were female, and all university educated. Median time since death was three years (range 1-5 years). Participants provided feedback about aesthetics, technical issues, and content. Interview responses were mapped to the theoretical framework of acceptability.
Conclusion: Participants found the website a helpful resource, and acceptable when considering the theoretical framework of acceptability.
Practice implications: This website fills a current gap in the care of families after SCD; more work is needed to assess effectiveness.
... Patient education is one of the main professional health and therapeutic functions. It is a patient-centered process and based on patients' needs to help them make participatory (1). From the quality of care viewpoint, it is a part of patient's rights (2). ...
Introduction: Patient education is a dynamic and continuous process that starts from the moment of admitting the patient and continues after the discharge. The objective of such education is to empower patients to do self-care and improve their quality of lives. Different methods are available for education. This study was aimed to explore the Facilitators and barriers of remote education using mobile massaging applications. Materials and Methods: The study was conduct as a participatory action research. Participants were 70 nurses, 14 physicians, and 96 patients. Data was collect with interviews. Data analysis method was content analysis and the research process consisted of four phases of planning, implementing, observing, and reflecting. Results: The categories of the Facilitators of mobile learning from physicians, nurses, and patients’ viewpoints were “more accessibility, complete and comprehensive, and ease of virtual education.” Among the barriers were “large volume of content, out of date content that puzzles the users, invalidity of some references, and the risk of infection transmission through mobile phone.” The participants emphasized on the necessity of codifying content, updating educational resources, teaching the methods of using reliable references, and codifying guidelines of disinfecting mobile phones. Conclusion: Patient education through remote education in virtual space using massaging applications is an efficient, comprehensive, accessible, and economic method that also brings patients’ satisfaction. It is essential to use experts’ viewpoints to update the references. The patients need to be educated about the right way of disinfecting mobile devices and find reliable references.
... Patient Education Materials Assessment Tool for print material (PEMAT-P): Understandability and Actionability-The PEMAT-P assesses understandability (information comprehension) and actionability (action assistance) of print items by evaluating components such as content, word choice and style, numbers, organization, layout and design, visuals, and ability to activate [49]. ...
Background:
Healthy People 2030 emphasizes personal health literacy (individual skills) and organizational health literacy-the degree to which organizations equitably enable individuals to find, understand, and use information and services to inform health-related decisions and actions for themselves and others. However, research on the latter is in the early stages.
Methods:
This study describes an organizational health literacy assessment in a U.S. urban academic children's hospital. A variety of evidence-based health literacy assessments were used to assess patient information materials and the environment, including institutional practices, navigation, culture and language, and communication. Trained interviewers and analysts reached consensus for all assessments.
Results:
Information Items: SMOG scores (n = 9) ranged from 7th- to 14th-grade reading level (average = 11.3). PEMAT-P scores (n = 9) ranged from 43.8% to 93.8% understandability and 0% to 80% actionability. CDC CCI scores (literacy demand) (n = 6) ranged from 18.2% to 58.8% (≥90% = excellent). SAM scores (suitability) (n = 6) fell in the "adequate" range (43.2-58.3%). The PMOSE/IKIRSCH scores (complexity) (n = 3) noted low-moderate difficulty. Apter's Hierarchy (n = 4) revealed three numeracy domains (50% = descriptive purposes and decision-making, 100% = interpreting information). Organization-level: Walking interviews highlighted organizational facilitators and barriers related to the pre-visit and visit environments. HLE2 domain scores ranged from 52% to 68%.
Conclusions:
Organizational health literacy demands far outweigh the average literacy skills of adults in the U.S. (patients and staff). Findings can be used to hone quality improvement and other processes to focus on structural solutions to increase health equity.
... The Patient Education Materials Assessment Tool (PEMAT) was used to evaluate the understandability (PEMAT-U) and actionability (PEMAT-A) of each video. 14, 15 For videos and other audiovisual materials, 11 criteria comprise PEMAT-U, and four comprise PEMAT-A. A two-point Likert scale is used to answer each question proposed regarding the educational material, where a "0" means disagreement, a "1" means agreement, and, where relevant, "Not Applicable" (NA) can also be used. ...
... However, it is worthwhile to note there are several validated, widely used instruments measuring constructs highly relevant to PEMs. Examples include readability algorithms (e.g., Flesch Reading Ease Formula, Flesch Kincaid Grade Level Formula, Gunning Fox Index, Coleman Liau Index, Simple Measure of Gobbledygook (SMOG), Automated Readability Index, and the Linsear Write Formula), health literacy/numeracy instruments (Short Assessment of Health Literacy (SAHL) [56], Rapid Estimate of Adult Health Literacy in Medicine (REALM) [57], rapid assessment of health literacy [58], Newest Vital Sign (NVS) [59], the System Usability Scale (SUS) [60], and the 'gold standard' Patient Education Materials Assessment Tool (PEMAT) from the U.S. Agency for Healthcare Quality and Research [61]. Employing validated tools helps limit bias in evaluation and can provide strong evidence that the materials are acceptable to patients. ...
Co-creating patient-facing educational materials (PEMs) can enhance person-centered care by responding to patient priorities and unmet needs. Little data exist on ‘best practices’ for co-creation. We followed the Arksey and O’Malley framework to conduct a systematic literature search of nine databases (MEDLINE, PubMed, EMBASE, CINAHL, PsycINFO, Web of Science, Cochrane Library, Joanna Briggs Institute, TRIP—April, 2022) to identify empirical studies published in English on PEM co-creation to distill ‘best practices’. Following an independent dual review of articles, data were collated into tables, and thematic analysis was employed to synthesize ‘best practices’ that were validated by a patient experienced in co-creating PEMs. Bias was not assessed, given the study heterogeneity. Of 6998 retrieved articles, 44 were included for data extraction/synthesis. Studies utilized heterogeneous methods spanning a range of health conditions/populations. Only 5/45 (11%) studies defined co-creation, 14 (32%) used a guiding framework, and 18 (41%) used validated evaluation tools. Six ‘best practices’ were identified: (1) begin with a review of the literature, (2) utilize a framework to inform the process, (3) involve clinical and patient experts from the beginning, (4) engage diverse perspectives, (5) ensure patients have the final decision, and (6) employ validated evaluation tools. This scoping review highlights the need for clear definitions and validated evaluation measures to guide and assess the co-creation process. Identified ‘best practices’ are relevant for use with diverse patient populations and health issues to enhance person-centered care.
... We chose the four dimensions after reviewing the literature and finding existing evaluation methods missed important considerations. For example, the Patient Education Materials Assessment Tool [34] focuses on understandability and actionability, clearly important but too narrowly focused for our purposes. We also sought to determine acceptability to a diverse group of participants and ensure that we could identify and minimize unintended harms. ...
Objectives
We evaluated the What Matters to Me Workbook, a patient-facing version of the Serious Illness Conversation Guide co-created by Ariadne Labs and The Conversation Project.
Methods
We purposively recruited diverse seriously ill patients and caregivers in the US. Participants completed the Workbook, a survey, and a semi-structured in-depth interview about their experience. Qualitative analysis of interviews and notes was employed to extract themes. Simple descriptive statistics were employed to analyze eight investigator authored questions.
Results
Twenty-nine study participants completed twenty-one interviews and twenty-five surveys. Ratings for safety (3.87/4, SD = 0.43) and acceptability (3.59/4, SD = 0.956) were higher than ratings for ease of use (3.30/4, SD = 0.97) and usefulness (3.24/4, SD = 0.80). Qualitative analysis identified that while the workbook was safe, acceptable, easy to use, and useful, it is more important who is recommending it and how they are explaining it.
Conclusion
If presented in the right way by a trustworthy person, the What Matters to Me Workbook can be an easy to use, useful, and safe resource for patients with serious illness and their caregivers.
Innovation
The Workbook focuses on serious illness rather than end-of-life and meshes with a clinician-facing conversation guide and a health-system level intervention.
... With the help of a graphic designer, we then transformed the text into a set of infographics that used color and graphics to convey pertinent information. The infographics also incorporated health literacy 'best practices' to enhance readability and understandability [18][19][20][21]. Specifically, we used plain language, short sections of text, and incorporated images to support learning. ...
Objective
Advancements in diagnostics and treatment options for cardiac amyloidosis have improved patient outcomes, yet few patient education materials exist to help patients understand the disease and diagnosis process. We sought to develop and evaluate a set of plain language, patient-centered infographics describing the condition and common diagnostic tests.
Methods
Using health literacy best practices, we developed 7 infographics which were further revised based on multilevel stakeholder feedback. To evaluate the materials, we recruited 100 patients from healthcare settings in Chicago, IL; participants completed a web-assisted interview during which they were randomized 1:1 to first view either our infographics or a standard material. Participants completed a knowledge assessment on their assigned material and subsequently reported impressions of both materials.
Results
No differences were found between study arms in knowledge. The infographics took significantly less time to read and were more highly rated by participants in terms of appearance and understandability. Over two-thirds of participants preferred the infographics to the standard.
Conclusions
The infographics created may improve the learning process about a complex condition and diagnosis process unknown to most adults.
Innovation
These infographics are the first of their kind for cardiac amyloidosis and were created using health literacy best practices.
Background
Printed educational materials (PEMs) have been used for patient education in various settings. The purpose of this study was to determine the readability, understandability, and actionability of trauma-related educational material from the Pediatric Orthopaedic Society of North America (POSNA, Orthokids), as well as determine its efficacy in educating pediatric orthopaedic trauma patients and caregivers.
Methods
The readability, understandability and actionability of PEMs was assessed using the Patient Education materials Assessment Tool (PEMAT). Five reviewers ranging in experience independently evaluated the educational materials. The efficacy of PEMs was assessed prospectively by randomizing patients into 2 groups. The first group (Education) received the OrthoKids educational material related to the patient’s fracture. The second group (No Education) did not receive the educational material. At the first follow-up visit, parents/guardians in both groups completed surveys. Statistical analyses included descriptive and univariate statistics.
Results
The understandability of PEMs was similar (68% to 74%); however, the educational materials had varying actionability scores ranging from 20% for femoral shaft fractures to 60% for elbow fractures. In total, 101 patients were randomized to assess the efficacy of educational materials (Education=51, No Education=50). There were no significant differences in sex, age, race/ethnicity, and level of education between caregivers in both groups ( P > 0.05). Only 61% (31/51) participants in the Education group reported using the educational material; however, 67% to 68% of participants in either group reported wanting PEMs. Participants in the group that did not receive PEMs were significantly more likely to use the internet to find more information (74% vs. 51%, P < 0.05).
Conclusions
This study suggests that participants that did not receive PEMs were significantly more likely to search the internet for more information. Improving the quality and actionability of educational resources on electronic platforms is needed to improve patient education. A multi-modal approach using PEMs that includes a list of high-quality online sources would likely be most effective in educating pediatric trauma patients and caregivers.
Level of Evidence
I
To enable recourse, explanations provided to people should be actionable, that is, explain what a person should do to change the model’s decision. However, what actionability means in the context of explainable AI is unclear. In this paper, we explore existing tools that others developed to evaluate actionability in their respective domains. To our knowledge, no prior work in the XAI field has developed such a tool to evaluate the actionability of explanation. We conducted an experimental study to validate two existing actionability tools for discriminating the actionability of two types of explanations. Our results indicate that the two existing actionability tools reveal metrics relevant for conceptualising actionability for the XAI community.
Introduction
Women with type 2 diabetes (T2DM) are more likely to experience adverse reproductive outcomes, yet preconception care can significantly reduce these risks. For women with T2DM, preconception care includes reproductive planning and patient education on: (1) the importance of achieving glycaemic control before pregnancy, (2) using effective contraception until pregnancy is desired, (3) discontinuing teratogenic medications if pregnancy could occur, (4) taking folic acid, and (5) managing cardiovascular and other risks. Despite its importance, few women with T2DM receive recommended preconception care.
Methods and analysis
We are conducting a two-arm, clinic-randomised trial at 51 primary care practices in Chicago, Illinois to evaluate a technology-based strategy to ‘hardwire’ preconception care for women of reproductive age with T2DM (the PREPARED (Promoting REproductive Planning And REadiness in Diabetes) strategy) versus usual care. PREPARED leverages electronic health record (EHR) technology before and during primary care visits to: (1) promote medication safety, (2) prompt preconception counselling and reproductive planning, and (3) deliver patient-friendly educational tools to reinforce counselling. Post-visit, text messaging is used to: (4) encourage healthy lifestyle behaviours. English and Spanish-speaking women, aged 18–44 years, with T2DM will be enrolled (N=840; n=420 per arm) and will receive either PREPARED or usual care based on their clinic’s assignment. Data will be collected from patient interviews and the EHR. Outcomes include haemoglobin A1c (primary), reproductive knowledge and self-management behaviours. We will use generalised linear mixed-effects models (GLMMs) to evaluate the impact of PREPARED on these outcomes. GLMMs will include a fixed effect for treatment assignment (PREPARED vs usual care) and random clinic effects.
Ethics and dissemination
This study was approved by the Northwestern University Institutional Review Board (STU00214604). Study results will be published in journals with summaries shared online and with participants upon request.
Trial registration number
ClinicalTrials.gov Registry ( NCT04976881 ).
Issue Addressed
Aboriginal and Torres Strait Islander women experience considerable disparities in gynaecological cancer outcomes. Accessible and culturally appropriate health promotion resources about gynaecological cancers may support health literacy in this area. This study aimed to determine the understandability, actionability, readability, and cultural relevance of gynaecological cancer health literacy resources for Aboriginal and Torres Strait Islander consumers, families, and caregivers.
Methods
We conducted a general Google search and targeted searches of Australian gynaecological cancer organisation websites in January and February 2022. Resources were assessed for understandability, actionability and cultural relevance to Aboriginal and Torres Strait Islander audiences.
Results
We found 16 resources. The resources were generally understandable, actionable, readable, and culturally relevant, however, most resources were focused on cervical cancer prevention through vaccination and screening. Few resources focused other gynaecological cancer types or aspects of the cancer care continuum. While many resources contained elements that made them culturally relevant, areas for improvement were identified. These included: greater transparency relating to the Aboriginal and Torres Strait Islander leadership, governance, and involvement in the development of the resources as well as availability of different resource formats with an emphasis on visual aids.
Conclusions
This study highlighted a need for the development of resources relating to a wider range of gynaecological cancer types and different stages of the cancer care continuum for Aboriginal and Torres Strait Islander women.
So What?
The development of a broader range of culturally appropriate gynaecological cancer health literacy resources, ideally developed through co‐design with Aboriginal and Torres Strait Islander peoples, may contribute to addressing the disparities in gynaecological cancer outcomes for Aboriginal and Torres Strait Islander women.
Objectives
This research aims at evaluating the quality of anal cancer and its precancerous lesions-related videos on YouTube.
Design
Cross-sectional survey design.
Setting
USA.
Participants
The top 150 videos on YouTube were selected for analysis based on three search terms. The duplicate, irrelevant, commercial, operation-related and audio-free videos were excluded. Finally, 105 relevant videos were included.
Methods
We assessed the completeness of video content from six dimensions and marked the men having sex with men (MSM)-related videos. To measure the video quality, DISCERN, Journal of the American Medical Association ( JAMA ) Benchmark Criteria, Patient Education Materials Assessment Tool (PEMAT) and Global Quality Scale (GQS) were used. The correlation between DISCERN classification and duration, JAMA , PEMAT and GQS scores were recorded.
Results
The video content was mainly about the management of the disease (mean score 1.086). Overall, the quality of videos uploaded by the non-profit organisation was relatively high. A correlation existed between each other of the JAMA , DISCERN and GQS scores (p<0.001). Moreover, they were positively correlated with video duration and PEMAT scores (p<0.001).
Conclusions
Although the overall quality of information about anal cancer and its precancerous lesions videos on YouTube is acceptable, it might not fully meet the health information needs of patients. Therefore, they should exercise caution when using YouTube as a source of anal cancer-related information, especially the MSM population.
Background
Social platforms such as YouTube have become sources of information about diseases as they can be easily and rapidly accessed. However, this also has the risk of ill-intentioned content and misleading information.
Objective
To evaluate the reliability of YouTube video content about delayed ejaculation treatment.
Material and methods
YouTube videos were searched using the terms “delayed ejaculation,” “retarded ejaculation,” “inhibited ejaculation,” and “anejaculation.” Videos were excluded if they were not in English, were not related to the subject, or did not have audio and visual content. In accordance with the scientifically proven accurate information, the videos were separated as reliable (Group 2, n: 112) and unreliable videos (Group 1, n: 94). The groups were compared in respect of the video characteristics, and the scores obtained in the DISCERN-5, Global Quality Scale, the Patient Education Materials Assessment Tool Audiovisual, and the Journal of the American Medical Association scales. Intraclass correlation test was used to evaluate the level of agreement between the two investigators.
Results
Of the 1200 videos, 994 were excluded. No significant difference was determined between the Group 1 and Group 2 in respect of the median number of views [1672 (4555) vs 1547 (28,559), p = 0.63] and likes [10 (42) vs 17 (255), p = 0.07]. There was a greater number of videos in the Group 2 (54.4%) and the points obtained on the scoring scales were significantly higher than the Group 1 ( p < 0.001). The videos originating from universities/professional organizations/non-profit physician/physician group were comprised the majority of the reliable videos (55.3%) and the unreliable videos had more content related to treatment (71.4%) ( p < 0.001).
Conclusion
Although there was a greater number of reliable videos related to the problem of delayed ejaculation, the content could be misleading and should be avoided by patients seeking treatment without consulting a physician.
Purpose of the review
Effective education enables people to modify the distress and impact of breathlessness by integrating evidence-informed breathlessness-related behaviours (knowledge, skill, attitude) into everyday life. This review considers recent studies of educational approaches focussed on chronic breathlessness as a modifiable, noxious and debilitating multidimensional experience.
Recent findings
Systematic assessments of text-based patient education materials and mobile phone applications specific to breathlessness indicate that while these resources are readily available, issues continue to persist with quality, readability, usefulness and availability of non-English language versions. Various forms of educational interventions for breathlessness have proven feasible and valued by people living with breathlessness and their significant others (uptake/completion, personal benefit, ripple effect on health professionals). Health professional knowledge about the impact of chronic breathlessness and effective management can be altered through structured, educational interventions.
Summary
Empiric studies of patient education for breathlessness are scarce despite persistent calls for better breathlessness education for people living with or providing care for someone living with this noxious symptom. In clinical practice, it is highly likely that there are effective and ineffective educational practices, both of which, if publicly disseminated, would inform future educational strategies to advance breathlessness self-management.
Since its release at the end of 2022, the social response to ChatGPT, a large language model (LLM), has been huge, as it has revolutionized the way we communicate with computers. This review was performed to describe the technical background of LLMs and to provide a review of the current literature on LLMs in the field of oral and maxillofacial surgery (OMS). The PubMed, Scopus, and Web of Science databases were searched for LLMs and OMS. Adjacent surgical disciplines were included to cover the entire literature, and records from Google Scholar and medRxiv were added. Out of the 57 records identified, 37 were included; 31 (84%) were related to GPT-3.5, four (11%) to GPT-4, and two (5%) to both. Current research on LLMs is mainly limited to research and scientific writing, patient information/communication, and medical education. Classic OMS diseases are underrepresented. The current literature related to LLMs in OMS has a limited evidence level. There is a need to investigate the use of LLMs scientifically and systematically in the core areas of OMS. Although LLMs are likely to add value outside the operating room, the use of LLMs raises ethical and medical regulatory issues that must first be addressed.
Background:
Young women with metastatic breast cancer (YWMBC) are part of a digitally connected generation yet are underserved in terms of information needs. YouTube is widely used to find and identify health information. The accessibility of health-related content on social media, paired with the rare and marginalized experience of YWMBC, as well as the digital media practices of younger generations, implies a considerable likelihood that YWMBC will seek information and community online.
Objective:
This study aims to assess the content quality of MBC YouTube videos and to identify themes in YWMBC experiences based on YouTube videos.
Methods:
A systematic assessment of MBC YouTube videos with search terms "metastatic breast cancer young" was conducted in 08/2021. The search was performed in an incognito browser and with no associated YouTube or Google account. Search results were placed in order from most to least number of views. Title, date uploaded, length, poster identity, number of likes and comments were collected. Understandability, actionability were assessed using the Patient Education Materials Assessment Tool (PEMAT); information reliability/quality was assessed with DISCERN. Themes, sponsorships, healthcare professionals' and patients' narratives were reported.
Results:
A total of 101 videos were identified. Of these, 78.2% (79/101) included sponsorships. Mean PEMAT score was 78.8% (SD 15.3) and 43.1% (SD 45.2) for understandability and actionability, respectively. Mean DISCERN score was 2.44/5 (SD 0.7). Identified themes include treatment (66.3%; 67/101), family relationship (45.5%; 46/101), motherhood (37.6%; 38/101).
Conclusions:
YouTube videos about YWMBC are highly understandable but demonstrate moderate rates of actionability, with low reliability and quality scores. Many have a commercial bias. While online materials have limitations, their potential to provide patient support is not fully developed. By acknowledging their patients' media habits, healthcare professionals can 1) further develop a trusting bond with their patients, 2) provide a space for open and honest discussions of online materials, and 3) avoid any potential instances of confusion caused by misleading, inaccurate, or false online materials.
Objective:
To quantitatively compare online patient education materials found using traditional search engines (Google) versus conversational Artificial Intelligence (AI) models (ChatGPT) for benign paroxysmal positional vertigo (BPPV).
Study design:
The top 30 Google search results for "benign paroxysmal positional vertigo" were compared to the OpenAI conversational AI language model, ChatGPT, responses for 5 common patient questions posed about BPPV in February 2023. Metrics included readability, quality, understandability, and actionability.
Setting:
Online information.
Methods:
Validated online information metrics including Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease (FRE), DISCERN instrument score, and Patient Education Materials Assessment Tool for Printed Materials were analyzed and scored by reviewers.
Results:
Mean readability scores, FKGL and FRE, for the Google webpages were 10.7 ± 2.6 and 46.5 ± 14.3, respectively. ChatGPT responses had a higher FKGL score of 13.9 ± 2.5 (P < .001) and a lower FRE score of 34.9 ± 11.2 (P = .005), both corresponding to lower readability. The Google webpages had a DISCERN part 2 score of 25.4 ± 7.5 compared to the individual ChatGPT responses with a score of 17.5 ± 3.9 (P = .001), and the combined ChatGPT responses with a score of 25.0 ± 0.9 (P = .928). The average scores of the reviewers for all ChatGPT responses for accuracy were 4.19 ± 0.82 and 4.31 ± 0.67 for currency.
Conclusion:
The results of this study suggest that the information on ChatGPT is more difficult to read, of lower quality, and more difficult to comprehend compared to information on Google searches.
Importance:
Consumers are increasingly using artificial intelligence (AI) chatbots as a source of information. However, the quality of the cancer information generated by these chatbots has not yet been evaluated using validated instruments.
Objective:
To characterize the quality of information and presence of misinformation about skin, lung, breast, colorectal, and prostate cancers generated by 4 AI chatbots.
Design, setting, and participants:
This cross-sectional study assessed AI chatbots' text responses to the 5 most commonly searched queries related to the 5 most common cancers using validated instruments. Search data were extracted from the publicly available Google Trends platform and identical prompts were used to generate responses from 4 AI chatbots: ChatGPT version 3.5 (OpenAI), Perplexity (Perplexity.AI), Chatsonic (Writesonic), and Bing AI (Microsoft).
Exposures:
Google Trends' top 5 search queries related to skin, lung, breast, colorectal, and prostate cancer from January 1, 2021, to January 1, 2023, were input into 4 AI chatbots.
Main outcomes and measures:
The primary outcomes were the quality of consumer health information based on the validated DISCERN instrument (scores from 1 [low] to 5 [high] for quality of information) and the understandability and actionability of this information based on the understandability and actionability domains of the Patient Education Materials Assessment Tool (PEMAT) (scores of 0%-100%, with higher scores indicating a higher level of understandability and actionability). Secondary outcomes included misinformation scored using a 5-item Likert scale (scores from 1 [no misinformation] to 5 [high misinformation]) and readability assessed using the Flesch-Kincaid Grade Level readability score.
Results:
The analysis included 100 responses from 4 chatbots about the 5 most common search queries for skin, lung, breast, colorectal, and prostate cancer. The quality of text responses generated by the 4 AI chatbots was good (median [range] DISCERN score, 5 [2-5]) and no misinformation was identified. Understandability was moderate (median [range] PEMAT Understandability score, 66.7% [33.3%-90.1%]), and actionability was poor (median [range] PEMAT Actionability score, 20.0% [0%-40.0%]). The responses were written at the college level based on the Flesch-Kincaid Grade Level score.
Conclusions and relevance:
Findings of this cross-sectional study suggest that AI chatbots generally produce accurate information for the top cancer-related search queries, but the responses are not readily actionable and are written at a college reading level. These limitations suggest that AI chatbots should be used supplementarily and not as a primary source for medical information.
People indispensably use instructions shared by one another to work on unfamiliar tasks in daily or professional life. However, personally shared tutorials are often based on personal experiences and represent a collective overview of past encounters, which can be misaligned with the specific work context during the re-enacting time. Drawing inspiration from the effective dynamics observed in conversational instruction-giving and -following between experts and novices, we propose a chatbot system that delivers archived how-to tutorials by providing necessary information in a just-in-time manner, tailored to the needs of the instruction-follower. Our aim is to transform unaided instruction reading activities into conversational instruction-following experiences. We implemented a chatbot system and evaluated it through a between subject study. The results demonstrate a promise of leveraging human-chatbot interaction to support actionable instruction-following.Keywordschatbotinstruction-followinghow-to tutorialsituated action/plan
Nearly half of all American adults—90 million people—have difficulty understanding and acting upon health information. The examples below were selected from the many pieces of complex consumer health information used in America.
• From a research consent form: “A comparison of the effectiveness of educational media in combination with a counseling method on smoking habits is being examined.” (Doak et al., 1996)
• From a consumer privacy notice: “Examples of such mandatory disclosures include notifying state or local health authorities regarding particular communicable diseases.”
• From a patient information sheet: “Therefore, patients should be monitored for extraocular CMV infections and retinitis in the opposite eye, if only one infected eye is being treated.”
Forty million Americans cannot read complex texts like these at all, and 90 million have difficulty understanding complex texts. Yet a great deal of health information, from insurance forms to advertising, contains complex text. Even people with strong literacy skills may have trouble obtaining, understanding, and using health information: a surgeon may have trouble helping a family member with Medicare forms, a science teacher may not understand information sent by a doctor about a brain function test, and an accountant may not know when to get a mammogram.
This report defines health literacy as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” (Ratzan and Parker, 2000). However, health literacy goes beyond the individual obtaining information. Health literacy emerges when the expectations, preferences, and skills of individuals seeking health information and services meet the expectations, preferences, and skills of those providing information and services. Health literacy arises from a convergence of education, health services, and social and cultural factors. Although causal relationships between limited health literacy and health outcomes are not yet established, cumulative and consistent findings suggest such a causal connection.
Approaches to health literacy bring together research and practice from diverse fields. This report examines the body of knowledge in this emerging field, and recommends actions to promote a health-literate society. Increasing knowledge, awareness, and responsiveness to health literacy among health services providers as well as in the community would reduce problems of limited health literacy. This report identifies key roles for the Department of Health and Human Services as well as other public and private sector organizations to foster research, guide policy development, and stimulate the development of health literacy knowledge, measures, and approaches. These organizations have a unique and critical opportunity to ensure that health literacy is recognized as an essential component of high-quality health services and health communication.
Consideration of categories related to reading comprehension--beyond reading level--is imperative to reach low literacy populations effectively. "Suitability" has been proposed as a term to encompass six categories of such factors: content, literacy demand graphics, layout/typography, learning stimulation, and cultural appropriateness. Our purpose was to describe instruments used to evaluate categories of suitability in cancer education materials in published reports and their findings. We searched databases and reference lists for evaluations of print and Web-based cancer education materials to identify and describe measures of these categories. Studies had to evaluate reading level and at least one category of suitability. Eleven studies met our criteria. Seven studies reported inter-rater reliability. Cultural appropriateness was most often assessed; four instruments assessed only surface aspects of cultural appropriateness. Only two of seven instruments used, the suitability assessment of materials (SAM) and the comprehensibility assessment of materials (SAM + CAM), were described as having any evidence of validity. Studies using Simplified Measure of Goobledygook (SMOG) and Fry reported higher average reading level scores than those using Flesh-Kincaid. Most materials failed criteria for reading level and cultural appropriateness. We recommend more emphasis on the categories of suitability for those developing cancer education materials and more study of these categories and reliability and validity testing of instruments.
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Approximately 80 million Americans have limited health literacy, which puts them at greater risk for poorer access to care and poorer health outcomes.
To update a 2004 systematic review and determine whether low health literacy is related to poorer use of health care, outcomes, costs, and disparities in health outcomes among persons of all ages.
English-language articles identified through MEDLINE, CINAHL, PsycINFO, ERIC, and Cochrane Library databases and hand-searching (search dates for articles on health literacy, 2003 to 22 February 2011; for articles on numeracy, 1966 to 22 February 2011).
Two reviewers independently selected studies that compared outcomes by differences in directly measured health literacy or numeracy levels.
One reviewer abstracted article information into evidence tables; a second reviewer checked information for accuracy. Two reviewers independently rated study quality by using predefined criteria, and the investigative team jointly graded the overall strength of evidence.
96 relevant good- or fair-quality studies in 111 articles were identified: 98 articles on health literacy, 22 on numeracy, and 9 on both. Low health literacy was consistently associated with more hospitalizations; greater use of emergency care; lower receipt of mammography screening and influenza vaccine; poorer ability to demonstrate taking medications appropriately; poorer ability to interpret labels and health messages; and, among elderly persons, poorer overall health status and higher mortality rates. Poor health literacy partially explains racial disparities in some outcomes. Reviewers could not reach firm conclusions about the relationship between numeracy and health outcomes because of few studies or inconsistent results among studies.
Searches were limited to articles published in English. No Medical Subject Heading terms exist for identifying relevant studies. No evidence concerning oral health literacy (speaking and listening skills) and outcomes was found.
Low health literacy is associated with poorer health outcomes and poorer use of health care services.
Agency for Healthcare Research and Quality.
Health literacy requires reading and writing skills as well as knowledge of health topics and health systems. Materials written at high reading levels with ambiguous, technical, or dense text, often place great comprehension demands on consumers with lower literacy skills. This study developed and used an instrument to analyze cervical cancer prevention materials for readability, comprehensibility, suitability, and message design.
The Suitability Assessment of Materials (SAM) was amended for ease of use, inclusivity, and objectivity with the encouragement of the original developers. Other novel contributions were specifically related to "comprehensibility" (CAM). The resulting SAM + CAM was used to score 69 materials for content, literacy demand, numeric literacy, graphics, layout/typography, and learning stimulation variables. Expert reviewers provided content validation. Inter-rater reliability was "substantial" (kappa = .77).
The mean reading level of materials was 11th grade. Most materials (68%) scored as "adequate" for comprehensibility, suitability, and message design; health education brochures scored better than other materials. Only one-fifth were ranked "superior" for ease of use and comprehensibility.
Most written materials have a readability level that is too high and require improvement in ease of use and comprehensibility for the majority of readers.
In a fourfold table showing binary agreement of two observers, the observed proportion of agreement, p0, can be paradoxically altered by the chance-corrected ratio that creates kappa as an index of concordance. In one paradox, a high value of p0 can be drastically lowered by a substantial imbalance in the table's marginal totals either vertically or horizontally. In the second pardox, kappa will be higher with an asymmetrical rather than symmetrical imbalanced in marginal totals, and with imperfect rather than perfect symmetry in the imbalance. An adjustment that substitutes kappa max for kappa does not repair either problem, and seems to make the second one worse.
Pi (pi) and kappa (kappa) statistics are widely used in the areas of psychiatry and psychological testing to compute the extent of agreement between raters on nominally scaled data. It is a fact that these coefficients occasionally yield unexpected results in situations known as the paradoxes of kappa. This paper explores the origin of these limitations, and introduces an alternative and more stable agreement coefficient referred to as the AC1 coefficient. Also proposed are new variance estimators for the multiple-rater generalized pi and AC1 statistics, whose validity does not depend upon the hypothesis of independence between raters. This is an improvement over existing alternative variances, which depend on the independence assumption. A Monte-Carlo simulation study demonstrates the validity of these variance estimators for confidence interval construction, and confirms the value of AC1 as an improved alternative to existing inter-rater reliability statistics.
It is estimated that 1 out of 5 American adults lacks the literacy skills needed to function effectively in todays society and as patients they present a challenge to health instruction. This book provides teaching strategies to aid in making self-help and self-management a reality for the patient with low literacy skills. The magnitude of the problem is explored and myths about illiteracy are exemplified. Two chapters deal with comprehension of written materials from theory to practical proven testing techniques including the cloze technique and the word-recognition test. Another chapter offers 2 formulas for testing the readability of written materials: the SMOG formula and the Fry index. Once a problem is identified a 4-step teaching strategy gives tips on teaching the low-literacy patient. For those pamphlets and booklets commonly given to patients suggestions for writing and rewriting to simplify the text are given along with examples of comprehension level. The use of audiotapes for teaching is explored in depth for those who learn better by listening. A chapter on the use of visuals stresses simplification to reduce the amount of text reading to emphasize instruction and to motivate and appropriate examples are given. Where written material audiotapes and visuals are drafted and produced according to the guidelines in the book the last chapter is devoted to pretesting to determine whether intended concepts are coming through. It defines the components of pretesting describes procedures discusses possible stages for production and provides examples of results. Although the book is aimed at providing teaching methods for the low-literate patient the suggestions also apply to the learning disabled.
There is no consensus on how best to assess the health literacy demands of health information materials. Comprehensive, reliable, and valid assessment tools are needed. The authors report on the development, refinement, and testing of Health Literacy INDEX, a new tool reflecting empirical evidence and best practices. INDEX is comprised of 63 indicators organized into 10 criteria: plain language, clear purpose, supporting graphics, user involvement, skill-based learning, audience appropriateness, user instruction, development details, evaluation methods, and strength of evidence. In a sample of 100 materials, intercoder agreement was high: 90% or better for 52% of indicators, and above 80% for nearly all others. Overall scores generated by INDEX were highly correlated with average ratings from 12 health literacy experts (r = 0.89, p < .0001). Additional research is warranted to examine the association between evaluation ratings generated by INDEX and individual understanding, behaviors, and improved health. Health Literacy INDEX is a comprehensive tool with evidence for reliability and validity that can be used to evaluate the health literacy demands of health information materials. Although improvement in health information materials is just one aspect of mitigating the effects of limited health literacy on health outcomes, it is an essential step toward a more health literate public. [Supplementary materials are available for this article. Go to the publisher's online edition of Journal of Health Communication for the following free supplemental resource: Health Literacy INDEX].
OBJECTIVE: To develop a valid, reliable instrument to measure the functional health literacy of patients.
DESIGN: The Test of Functional Health Literacy in Adults (TOFHLA) was developed using actual hospital materials. The TOFHLA consists
of a 50-item reading comprehension and 17-item numerical ability test, taking up to 22 minutes to administer. The TOFHLA,
the Wide Range Achievement Test-Revised (WRAT-R), and the Rapid Estimate of Adult Literacy in Medicine (REALM) were administered
for comparison. A Spanish version was also developed (TOFHLA-S).
SETTING: Outpatient settings in two public teaching hospitals.
PATIENTS: 256 English- and 249 Spanish-speaking patients were approached. 78% of the English- and 82% of the Spanish-speaking patients
gave informed consent, completed a demographic survey, and took the TOFHLA or TOFHLA-S.
RESULTS: The TOFHLA showed good correlation with the WRAT-R and the REALM (correlation coefficients 0.74 and 0.84, respectively).
Only 52% of the English speakers completed more than 80% of the questions correctly. 15% of the patients could not read and
interpret a prescription bottle with instructions to take one pill by mouth four times daily, 37% did not understand instructions
to take a medication on an empty stomach, and 48% could not determine whether they were eligible for free care.
CONCLUSIONS: The TOFHLA is a valid, reliable indicator of patient ability to read health-related materials. Data suggest that a high proportion
of patients cannot perform basic reading tasks. Additional work is needed to determine the prevalence of functional health
illiteracy and its effect on the health care experience.
To examine the relationship between low health literacy and disease state control and between low health literacy medication adherence in the primary care setting.
The following databases were searched for relevant articles from date of inception to April 2008: The Cochrane Database of Systematic Reviews, Cumulative Index to Nursing & Allied Health Literature, EMBASE, Education Resources Information Center, PsycINFO, International Pharmaceutical Abstracts, and Iowa Drug Information Service. MEDLINE was searched from 1966 to April 2008. Key words included literacy, health literacy, health education, educational status, disease outcomes, health outcomes, adherence, medication adherence, and patient compliance. Additional articles were identified by reviewing reference sections of retrieved articles.
Studies using a validated measure of health literacy and performing statistical analysis to evaluate the relationship between health literacy and disease state control or medication adherence were evaluated.
Eleven evaluations, including 10 discrete studies, met eligibility criteria. Six studies evaluated the relationship between health literacy and disease state control, 3 evaluated health literacy and medication adherence, and 1 study evaluated health literacy and both outcomes. A quality rating of poor, fair, or good was assigned to each study based on the study question, population, outcome measures, statistical analysis, and results. Eight studies had good quality, 1 was fair, and 2 were poor. Two high-quality studies demonstrated statistically significant relationships with health literacy, 1 with disease state control and 1 with medication adherence. Limitations of the other studies included inadequate sample size, underrepresentation of patients with low health literacy, use of less objective outcome measures, and insufficient statistical analysis.
There may be a relationship between health literacy and disease state control and health literacy and medication adherence. Future research, with adequate representation of patients with low health literacy, is needed to further define this relationship and explore interventions to overcome the impact that low health literacy may have on patient outcomes.
This paper describes the development and psychometric properties of a Tool to Evaluate Materials Used in Patient Education (TEMPtEd), which was designed to assist healthcare professionals to evaluate and select printed patient educational materials for their clients.
Previously-developed instruments include attribute checklists, readability formulae and rating scales, but they have not been shown to be valid or reliable. The TEMPtEd is an attempt to overcome the limitations of previously developed instruments.
The instrument was developed using Strickland's framework, with pilot testing conducted from 2004 to 2007.
The overall ratings of a heart failure educational brochure between the TEMPtEd and the Suitability Assessment of Materials, a previously developed instrument, were not statistically significantly different. Statistically significant correlations were noted between the two instruments in the overall scale and four of the five subscales. The internal consistency of the TEMPtEd was 0.68; however, a reduction in the number of rating scale options resulted in an internal consistency of 0.83-0.84. Exploratory factor analysis identified a six-factor solution and accounted for 74% of the variance. Study participants preferred the TEMPtEd.
As a result of psychometric testing, the TEMPtEd appears to be a promising instrument for the evaluation of patient educational material.
The clarity of prescription drug instructions is a health literacy and medication safety concern.
To assess the variability of pharmacy interpretations of physician prescriptions.
Identically written prescriptions for 4 common medications (atorvastatin, alendronate, trimethoprim/sulfamethoxazole, ibuprofen) were filled in 6 pharmacies (2 largest chains, 2 grocery stores, 2 independents) in 4 cities (Boston, Chicago, Los Angeles, Austin).
Components of the instruction were coded as dose, frequency, administration route, timing, indication, and auxiliary instructions.
In all, 85 labels were evaluated. Dose frequency was omitted on 6% of instructions ("take 1 tablet for cholesterol"). Timing was explicitly stated on 2% of instructions ("in the morning"). All prescriptions included indications; pharmacies transcribed these onto 38% of labels. The prescription for alendronate stated not to lie down for at least 30 minutes after taking; this was transcribed with 50% of instructions. Reading difficulty was above recommended levels for 46% of instructions; with 14% greater than a high school level.
Efforts are needed to ensure patients receive clear, consistent information supporting safe medication use.
This paper presents a general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies. The procedure essentially involves the construction of functions of the observed proportions which are directed at the extent to which the observers agree among themselves and the construction of test statistics for hypotheses involving these functions. Tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interobserver agreement are developed as generalized kappa-type statistics. These procedures are illustrated with a clinical diagnosis example from the epidemiological literature.
An omnibus index offers a single summary expression for a fourfold table of binary concordance among two observers. Among the available other omnibus indexes, none offers a satisfactory solution for the paradoxes that occur with p0 and kappa. The problem can be avoided only by using ppos and pneg as two separate indexes of proportionate agreement in the observers' positive and negative decisions. These two indexes, which are analogous to sensitivity and specificity for concordance in a diagnostic marker test, create the paradoxes formed when the chance correction in kappa is calculated as a product of the increment in the two indexes and the increment in marginal totals. If only a single omnibus index is used to compared different performances in observer variability, the paradoxes of kappa are desirable since they appropriately "penalize" inequalities in ppos and pneg. For better understanding of results and for planning improvements in the observers' performance, however, the omnibus value of kappa should always be accompanied by separate individual values of ppos and pneg.
To develop a valid, reliable instrument to measure the functional health literacy of patients.
The Test of Functional Health Literacy in Adults (TOFHLA) was developed using actual hospital materials. The TOFHLA consists of a 50-item reading comprehension and 17-item numerical ability test, taking up to 22 minutes to administer. The TOFHLA, the Wide Range Achievement Test--Revised (WRAT-R), and the Rapid Estimate of Adult Literacy in Medicine (REALM) were administered for comparison. A Spanish version was also developed (TOFHLA-S).
Outpatient settings in two public teaching hospitals.
256 English- and 249 Spanish-speaking patients were approached. 78% of the English- and 82% of the Spanish-speaking patients gave informed consent, completed a demographic survey, and took the TOFHLA or TOFHLA-S.
The TOFHLA showed good correlation with the WRAT-R and the REALM (correlation coefficients 0.74 and 0.84, respectively). Only 52% of the English speakers completed more than 80% of the questions correctly. 15% of the patients could not read and interpret a prescription bottle with instructions to take one pill by mouth four times daily, 37% did not understand instructions to take a medication on an empty stomach, and 48% could not determine whether they were eligible for free care.
The TOFHLA is a valid, reliable indicator of patient ability to read health-related materials. Data suggest that a high proportion of patients cannot perform basic reading tasks. Additional work is needed to determine the prevalence of functional health illiteracy and its effect on the health care experience.
To develop a short instrument, called DISCERN, which will enable patients and information providers to judge the quality of written information about treatment choices. DISCERN will also facilitate the production of new, high quality, evidence-based consumer health information.
An expert panel, representing a range of expertise in consumer health information, generated criteria from a random sample of information for three medical conditions with varying degrees of evidence: myocardial infarction, endometriosis, and chronic fatigue syndrome. A graft instrument, based on this analysis, was tested by the panel on a random sample of new material for the same three conditions. The panel re-drafted the instrument to take account of the results of the test. The DISCERN instrument was finally tested by a national sample of 15 information providers and 13 self help group members on a random sample of leaflets from 19 major national self help organisations. Participants also completed an 8 item questionnaire concerning the face and content validity of the instrument.
Chance corrected agreement (weighted kappa) for the overall quality rating was kappa = 0.53 (95% CI kappa = 0.48 to kappa = 0.59) among the expert panel, kappa = 0.40 (95% CI kappa = 0.36 to kappa = 0.43) among information providers, and kappa = 0.23 (95% CI kappa = 0.19 to kappa = 0.27) among self help group members. Higher agreement levels were associated with experience of using the instrument and with professional knowledge of consumer health information. Levels of agreement varied across individual items on the instrument, reflecting the need for subjectivity in rating certain criteria. The trends in levels of agreement were similar among all groups. The final instrument consisted of 15 questions plus an overall quality rating. Responses to the questionnaire after the final testing revealed the instrument to have good face and content validity and to be generally applicable.
DISCERN is a reliable and valid instrument for judging the quality of written consumer health information. While some subjectivity is required for rating certain criteria, the findings demonstrate that the instrument can be applied by experienced users and providers of health information to discriminate between publications of high and low quality. The instrument will also be of benefit to patients, though its use will be improved by training.
To review the relationship between literacy and health outcomes.
We searched MEDLINE, Cumulative Index to Nursing and Allied Health (CINAHL), Educational Resources Information Center (ERIC), Public Affairs Information Service (PAIS), Industrial and Labor Relations Review (ILLR), PsychInfo, and Ageline from 1980 to 2003.
We included observational studies that reported original data, measured literacy with any valid instrument, and measured one or more health outcomes. Two abstractors reviewed each study for inclusion and resolved disagreements by discussion.
One reviewer abstracted data from each article into an evidence table; the second reviewer checked each entry. The whole study team reconciled disagreements about information in evidence tables. Both data extractors independently completed an 11-item quality scale for each article; scores were averaged to give a final measure of article quality.
We reviewed 3,015 titles and abstracts and pulled 684 articles for full review; 73 articles met inclusion criteria and, of those, 44 addressed the questions of this report. Patients with low literacy had poorer health outcomes, including knowledge, intermediate disease markers, measures of morbidity, general health status, and use of health resources. Patients with low literacy were generally 1.5 to 3 times more likely to experience a given poor outcome. The average quality of the articles was fair to good. Most studies were cross-sectional in design; many failed to address adequately confounding and the use of multiple comparisons.
Low literacy is associated with several adverse health outcomes. Future research, using more rigorous methods, will better define these relationships and guide developers of new interventions.
OBJECTIVE: To review the relationship between literacy and health outcomes.
DATA SOURCES: We searched MEDLINE, Cumulative Index to Nursing and Allied Health (CINAHL), Educational Resources Information Center (ERIC), Public Affairs Information Service (PAIS), Industrial and Labor Relations Review (ILLR), PsychInfo, and Ageline from 1980 to 2003.
STUDY SELECTION: We included observational studies that reported original data, measured literacy with any valid instrument, and measured one or more health outcomes. Two abstractors reviewed each study for inclusion and resolved disagreements by discussion.
DATA EXTRACTION: One reviewer abstracted data from each article into an evidence table; the second reviewer checked each entry. The whole study team reconciled disagreements about information in evidence tables. Both data extractors independently completed an 11-item quality scale for each article; scores were averaged to give a final measure of article quality.
DATA SYNTHESIS: We reviewed 3,015 titles and abstracts and pulled 684 articles for full review; 73 articles met inclusion criteria and, of those, 44 addressed the questions of this report. Patients with low literacy had poorer health outcomes, including knowledge, intermediate disease markers, measures of morbidity, general health status, and use of health resources. Patients with low literacy were generally 1.5 to 3 times more likely to experience a given poor outcome. The average quality of the articles was fair to good. Most studies were cross-sectional in design; many failed to address adequately confounding and the use of multiple comparisons.
CONCLUSIONS: Low literacy is associated with several adverse health outcomes. Future research, using more rigorous methods, will better define these relationships and guide developers of new interventions.
Adequate functional literacy skills positively influence individuals' ability to take control of their health. Print and Web-based cancer information is often written at difficult reading levels. This systematic review evaluates readability instruments (FRE, F-K, Fog, SMOG, Fry) used to assess print and Web-based cancer information and word recognition and comprehension tests (Cloze, REALM, TOFHLA, WRAT) that measure people's health literacy. Articles on readability and comprehension instruments explicitly used for cancer information were assembled by searching MEDLINE and Psyc INFO from 1993 to 2003. In all, 23 studies were included; 16 on readability, 6 on comprehension, and 1 on readability and comprehension. Of the readability investigations, 14 focused on print materials, and 2 assessed Internet information. Comprehension and word recognition measures were not applied to Web-based information. None of the formulas were designed to determine the effects of visuals or design factors that could influence readability and comprehension of cancer education information.
Conclusions Health literacy is a complicated construct that depends on individual capacity to communicate and the demands posed by society
and the health care system. More comprehensive tests are needed to understand the gap between capacities and current demands
to help guide efforts to educate children and adults about health issues and to develop health-related information that more
of the general public can understand. For research, new instruments are needed that will measure individuals reading fluency
more precisely without posing an undue response burden. Computer-assisted testing, which selects items from a bank of possible
items according to a baseline-predicted reading ability and responses to previous questions, should allow more accurate measurement
of indiidual capacity without increasing the time required to complete testing. It remains unclear whether it is possible
to develop an accurate, practical “screening” test to identify individuals with limited health literacy. Even if this goal
is achieved, it remains unclear whether it is better to screen patients or to adopt “universal precautions” to avoid miscommunication
by using plain language in all oral and written communication and confirming understanding with all patients by having them
repeat back their understanding of their diagnosis and treatment plan. George Bernard Shaw said, “The main problem with communication
is the assumption that it has occurred.” This is a universal truth that transcends reading ability.
Adverse events resulting from medication error are a serious concern. Patients' literacy and their ability to understand medication information are increasingly seen as a safety issue.
To examine whether adult patients receiving primary care services at a public hospital clinic were able to correctly interpret commonly used prescription medication warning labels.
In-person structured interviews with literacy assessment.
Public hospital, primary care clinic.
A total of 251 adult patients waiting for an appointment at the Louisiana State University Health Sciences Center in Shreveport (LSUHSC-S) Primary Care Clinic.
Correct interpretation, as determined by expert panel review of patients' verbatim responses, for each of 8 commonly used prescription medication warning labels.
Approximately one-third of patients (n=74) were reading at or below the 6th-grade level (low literacy). Patient comprehension of warning labels was associated with one's literacy level. Multistep instructions proved difficult for patients across all literacy levels. After controlling for relevant potential confounding variables, patients with low literacy were 3.4 times less likely to interpret prescription medication warning labels correctly (95% confidence interval: 2.3 to 4.9).
Patients with low literacy had difficulty understanding prescription medication warning labels. Patients of all literacy levels had better understanding of warning labels that contained single-step versus multiple-step instructions. Warning labels should be developed with consumer participation, especially with lower literate populations, to ensure comprehension of short, concise messages created with familiar words and recognizable icons.
How to write easy-to-read health materials
- S J Shoemaker
National Institutes of Health, U.S. National Library of Medicine. How to write easy-to-read health materials; 2013, Available from http://www.nlm.nih.gov/ medlineplus/etr.html [accessed 23.08.13]. S.J. Shoemaker et al. / Patient Education and Counseling 96 (2014) 395–403
The toolkit for making written material clear and effective Available from http://www.cms.gov/Writ-tenMaterialsToolkit
- Centers Services
Centers for Medicare & Medicaid Services. The toolkit for making written material clear and effective; 2012, Available from http://www.cms.gov/Writ-tenMaterialsToolkit/ [accessed 23.08.13].
Reviewer's guide to evaluating health information materials
- S Smith
Smith S. Reviewer's guide to evaluating health information materials; 2008,
Available
from
http://www.beginningsguides.com/upload/Reviewers-Guide.pdf [accessed 23.08.13].
The Lexile Framework
- Stenner Aj
- I Horabin
- Smith
- Dr
- Smith
Stenner AJ, Horabin I, Smith DR, Smith M. The Lexile Framework. Durham, NC: MetaMetrics; 1988.
Objective 11-2. Improvement of health literacy. In: Communicating health: priorities and strategies for progress
- R Rudd
Rudd R. Objective 11-2. Improvement of health literacy. In: Communicating
health: priorities and strategies for progress. Washington, DC: U.S. Department of Health and Human Services; 2003.
Teaching patients with low literacy skills Philadelphia: Lippincott
- Cc Doak
- Lg Doak
- Jh Root
Doak CC, Doak LG, Root JH. Teaching patients with low literacy skills. 2nd ed.
Philadelphia: Lippincott; 1996.
DISCERN: an instrument for judging the quality of written consumer health information on treatment choices
- D Charnock
- S Shepperd
- G Needham
- R Gann
Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for
judging the quality of written consumer health information on treatment
choices. J Epidemiol Commun Health 1999;53:105-11.
The Lexile Framework
- A J Stenner
- I Horabin
- D R Smith
- M Smith
Stenner AJ, Horabin I, Smith DR, Smith M. The Lexile Framework. Durham, NC:
MetaMetrics; 1988.
Variability in pharmacy interpretations of physician prescriptions
- M S Wolf
- P Shekelle
- N K Choudry
- J Agnew-Blais
- R M Parker
- W H Shrank
Wolf MS, Shekelle P, Choudry NK, Agnew-Blais J, Parker RM, Shrank WH.
Variability in pharmacy interpretations of physician prescriptions. Med Care
2009;47:370-3.