Figure 1 - uploaded by Adam Wierzbicki
Content may be subject to copyright.
The distribution of credibility ratings for the experts and for the lay users in all the experimental conditions.
Source publication
In this paper we present a study on the credibility of lay users evaluations of health related web content. We investigate the differences between their approach and the approach of medical experts, analyse whether we can increase their accuracy using a simple support system, and explore the effectiveness of the wisdom of crowds approach. We find t...
Context in source publication
Similar publications
In this research work, the latest version 7 (V7) of the Tropical Rainfall Measuring Mission (TRMM) Multi-satellite Precipitation Analysis (TMPA) 3B42 product is evaluated over Greece, at five different temporal scales; 3, 6, 12, 24 and 48 hrs. Evaluation consists in assessing the performance of the product in reproducing observed, non-zero precipit...
Evaluating the accuracy (ie, estimating the sensitivity and specificity) of new diagnostic tests without the presence of a gold standard is of practical meaning and has been the subject of intensive study for several decades. Existing methods use 2 or more diagnostic tests under several basic assumptions and then estimate the accuracy parameters vi...
Citations
... Distinguishing between credible and non-credible online medical information poses a substantial challenge even for experienced medical professionals, and even more so for ordinary Web users whose evaluation may be impacted by cognitive biases or psychological factors [1,34]. Labeling source websites as either credible or non-credible is insufficient since false claims can be a part of an article originating from a credible source and vice versa. ...
... The dataset is open-sourced and publicly available. 1 Nine medical experts took part in the experiment, including 2 cardiologists, 1 gynecologist, 3 psychiatrists, and 3 pediatricians. All experts have completed 6-years medical studies and then a 5-year residency program. ...
Fighting medical disinformation in the era of the pandemic is an increasingly important problem. Today, automatic systems for assessing the credibility of medical information do not offer sufficient precision, so human supervision and the involvement of medical expert annotators are required. Our work aims to optimize the utilization of medical experts’ time. We also equip them with tools for semi-automatic initial verification of the credibility of the annotated content. We introduce a general framework for filtering medical statements that do not require manual evaluation by medical experts, thus focusing annotation efforts on non-credible medical statements. Our framework is based on the construction of filtering classifiers adapted to narrow thematic categories. This allows medical experts to fact-check and identify over two times more non-credible medical statements in a given time interval without applying any changes to the annotation flow. We verify our results across a broad spectrum of medical topic areas. We perform quantitative, as well as exploratory analysis on our output data. We also point out how those filtering classifiers can be modified to provide experts with different types of feedback without any loss of performance.
... Distinguishing between reliable and unreliable online health information poses a substantial challenge for lay Internet users [1]. Labeling source websites as either credible or non-credible is not sufficient as false claims can be a part of an article originating from a credible source and vice versa. ...
Fighting medical disinformation in the era of the global pandemic is an increasingly important problem. As of today, automatic systems for assessing the credibility of medical information do not offer sufficient precision to be used without human supervision, and the involvement of medical expert annotators is required. Thus, our work aims to optimize the utilization of medical experts’ time. We use the dataset of sentences taken from online lay medical articles. We propose a general framework for filtering medical statements that do not need to be manually verified by medical experts. The results show the gain in fact-checking performance of expert annotators on capturing misinformation by the factor of 2.2 on average. In other words, our framework allows medical experts to fact-check and identify over two times more non-credible medical statements in a given time interval without applying any changes to the annotation flow.
... Companies such as Google aim to discern the veracity of statements of fact contained in webpages. 1 Crowdsourced services striving to filter out non-credible information have been subject to research and are applied in practice. Among systems using that approach are the Article Feedback Tool on Wikipedia, the TweetCred 2 system for Twitter, or the WOT system for evaluating Web portal credibility. ...
... The evaluation presented in this section is based on an experiment [1]. A single task in this experiment consisted of evaluations of three websites drawn randomly from a corpus of 190 webpages on nine health-related topics. ...
... Corresponding search phrases were formulated, each aimed at finding both credible and not credible information. Random websites were selected from those that were shown on the first results pages (as searched for in the USA), and Source: [1] those that did not contain any meaningful content that could be subject to credibility evaluation were filtered out. The list of medical topics used in the study together with the number of webpages and the number of lay user evaluations for each topic is presented in Table 3.3. ...
This chapter creates a groundwork and conceptual foundation for the rest of the book. The chapter introduces a definition of information credibility and of Web content credibility evaluation support. Methods of measuring Web content credibility are discussed, along with available datasets, including fake news datasets. The subject of bias and subjectivity of Web content credibility evaluations is discussed.
... Michael Tomasello Credibility has recently become a hot topic in Web content research. Companies such as Google aim to discern the veracity of statements of fact contained in Web pages 1 . Crowdsourced services striving to filter out non-credible information have been subject to research and are applied in practice. ...
... The evaluation presented in this section is based on an experiment [1]. A single task in this experiment consisted of evaluations of three websites drawn randomly from a corpus of 190 webpages on 9 health-related topics. ...
... The lay user data presented in this section is based on 3 experimental treatments that were part of the experimental study [1]. Participants were assigned to the treatments randomly. ...
This chapter deals with methods and algorithms of Web content credibility evaluation support. The chapter introduces several examples of credibility evaluation support (CS) systems. A reference design of a CS system guides the reader. Various algorithms that are part of the CS system are discussed. Algorithms for automatic evaluation of statement credibility are discussed.
... Michael Tomasello Credibility has recently become a hot topic in Web content research. Companies such as Google aim to discern the veracity of statements of fact contained in Web pages 1 . Crowdsourced services striving to filter out non-credible information have been subject to research and are applied in practice. ...
... The evaluation presented in this section is based on an experiment [1]. A single task in this experiment consisted of evaluations of three websites drawn randomly from a corpus of 190 webpages on 9 health-related topics. ...
... The lay user data presented in this section is based on 3 experimental treatments that were part of the experimental study [1]. Participants were assigned to the treatments randomly. ...
This book introduces readers to Web content credibility evaluation and evaluation support. It highlights empirical research and establishes a solid foundation for future research by presenting methods of supporting credibility evaluation of online content, together with publicly available datasets for reproducible experimentation, such as the Web Content Credibility Corpus.
The book is divided into six chapters. After a general introduction in Chapter 1, including a brief survey of credibility evaluation in the social sciences, Chapter 2 presents definitions of credibility and related concepts of truth and trust. Next, Chapter 3 details methods, algorithms and user interfaces for systems supporting Web content credibility evaluation. In turn, Chapter 4 takes a closer look at the credibility of social media, exemplified in sections on Twitter, Q&A systems, and Wikipedia, as well as fake news detection. In closing, Chapter 5 presents mathematical and simulation models of credibility evaluation, before a final round-up of the book is provided in Chapter 6.
Overall, the book reviews and synthesizes the current state of the art in Web content credibility evaluation support and fake news detection. It provides researchers in academia and industry with both an incentive and a basis for future research and development of Web content credibility evaluation support services.
Misinformation on the Internet, deliberate or merely out of ignorance, is a serious problem and it puts users in the position of needing strong critical thinking skills to sort wheat from chaff. This book will help. It's an impressive exploration of ideas in the area of Web content credibility evaluation support. – Vint Cerf, Vice President and Chief Internet Evangelist at Google
... Michael Tomasello Credibility has recently become a hot topic in Web content research. Companies such as Google aim to discern the veracity of statements of fact contained in Web pages 1 . Crowdsourced services striving to filter out non-credible information have been subject to research and are applied in practice. ...
... The evaluation presented in this section is based on an experiment [1]. A single task in this experiment consisted of evaluations of three websites drawn randomly from a corpus of 190 webpages on 9 health-related topics. ...
... The lay user data presented in this section is based on 3 experimental treatments that were part of the experimental study [1]. Participants were assigned to the treatments randomly. ...
This chapter deals with credibility evaluation on Web-based social media, in particular, Twitter, Q&A systems, and Wikipedia. The special characteristics of credibility evaluation support on each of these platforms are discussed in detail, along with the most successful approaches of credibility evaluation support. Algorithms for fake news detection are reviewed in this chapter.
... Michael Tomasello Credibility has recently become a hot topic in Web content research. Companies such as Google aim to discern the veracity of statements of fact contained in Web pages 1 . Crowdsourced services striving to filter out non-credible information have been subject to research and are applied in practice. ...
... The evaluation presented in this section is based on an experiment [1]. A single task in this experiment consisted of evaluations of three websites drawn randomly from a corpus of 190 webpages on 9 health-related topics. ...
... The lay user data presented in this section is based on 3 experimental treatments that were part of the experimental study [1]. Participants were assigned to the treatments randomly. ...
This chapter introduces a theoretical model of Web content credibility that is based on the definition introduced in Chap. 2. The Credibility Game can be used to model reputed, surface, and earned credibility. The model is used to evaluate the effects of a reputation system (an important component of the CS system reference design introduced in Chap. 3) on the global behavior of content producers. The model can also be used to evaluate the effectiveness of methods for fake news detection.
... When over 2,500 users were asked to describe the features they actually used to determine website credibility, almost half mentioned "site presentation" as a key factor [12]. In general, users seem to have a positive evaluation bias, thinking that sites are more credible than they actually are [1]. However, with expert suggestions on the credibility of Web sources at their disposal, users are able to make better decisions [1]. ...
... In general, users seem to have a positive evaluation bias, thinking that sites are more credible than they actually are [1]. However, with expert suggestions on the credibility of Web sources at their disposal, users are able to make better decisions [1]. This idea of added context improving decision-making motivated our interest in investigating Knowledge Panels closely. ...
Recent research has suggested that young users are not particularly skilled in assessing the credibility of online content. A follow up study comparing students to fact checkers noticed that students spend too much time on the page itself, while fact checkers performed "lateral reading", searching other sources. We have taken this line of research one step further and designed a study in which participants were instructed to do lateral reading for credibility assessment by inspecting Google's search engine result page (SERP) of unfamiliar news sources. In this paper, we summarize findings from interviews with 30 participants. A component of the SERP noticed regularly by the participants is the so-called Knowledge Panel, which provides contextual information about the news source being searched. While this is expected, there are other parts of the SERP that participants use to assess the credibility of the source, for example, the freshness of top stories, the panel of recent tweets, or a verified Twitter account. Given the importance attached to the presence of the Knowledge Panel, we discuss how variability in its content affected participants' opinions. Additionally, we perform data collection of the SERP page for a large number of online news sources and compare them. Our results indicate that there are widespread inconsistencies in the coverage and quality of information included in Knowledge Panels.
... Michael Tomasello Credibility has recently become a hot topic in Web content research. Companies such as Google aim to discern the veracity of statements of fact contained in Web pages 1 . Crowdsourced services striving to filter out non-credible information have been subject to research and are applied in practice. ...
... The evaluation presented in this section is based on an experiment [1]. A single task in this experiment consisted of evaluations of three websites drawn randomly from a corpus of 190 webpages on 9 health-related topics. ...
... The lay user data presented in this section is based on 3 experimental treatments that were part of the experimental study [1]. Participants were assigned to the treatments randomly. ...
This book introduces readers to Web content credibility evaluation and evaluation support. It highlights empirical research and establishes a solid foundation for future research by presenting methods of supporting credibility evaluation of online content, together with publicly available datasets for reproducible experimentation, such as the Web Content Credibility Corpus.
The book is divided into six chapters. After a general introduction in Chapter 1, including a brief survey of credibility evaluation in the social sciences, Chapter 2 presents definitions of credibility and related concepts of truth and trust. Next, Chapter 3 details methods, algorithms and user interfaces for systems supporting Web content credibility evaluation. In turn, Chapter 4 takes a closer look at the credibility of social media, exemplified in sections on Twitter, Q&A systems, and Wikipedia, as well as fake news detection. In closing, Chapter 5 presents mathematical and simulation models of credibility evaluation, before a final round-up of the book is provided in Chapter 6.
Overall, the book reviews and synthesizes the current state of the art in Web content credibility evaluation support and fake news detection. It provides researchers in academia and industry with both an incentive and a basis for future research and development of Web content credibility evaluation support services.
Misinformation on the Internet, deliberate or merely out of ignorance, is a serious problem and it puts users in the position of needing strong critical thinking skills to sort wheat from chaff. This book will help. It's an impressive exploration of ideas in the area of Web content credibility evaluation support. – Vint Cerf, Vice President and Chief Internet Evangelist at Google
