Content uploaded by Lukasz Balwicki
Author content
All content in this area was uploaded by Lukasz Balwicki on Apr 07, 2021
Content may be subject to copyright.
Available via license: CC BY-NC-ND 4.0
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=khvi20
Human Vaccines & Immunotherapeutics
ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/khvi20
Vaccine misinformation on social media – topic-
based content and sentiment analysis of Polish
vaccine-deniers’ comments on Facebook
Krzysztof Klimiuk , Agnieszka Czoska , Karolina Biernacka & Łukasz Balwicki
To cite this article: Krzysztof Klimiuk , Agnieszka Czoska , Karolina Biernacka & Łukasz Balwicki
(2021): Vaccine misinformation on social media – topic-based content and sentiment analysis of
Polish vaccine-deniers’ comments on Facebook, Human Vaccines & Immunotherapeutics, DOI:
10.1080/21645515.2020.1850072
To link to this article: https://doi.org/10.1080/21645515.2020.1850072
© 2021 The Author(s). Published with
license by Taylor & Francis Group, LLC.
Published online: 30 Jan 2021.
Submit your article to this journal
Article views: 108
View related articles
View Crossmark data
RESEARCH PAPER
Vaccine misinformation on social media – topic-based content and sentiment analysis
of Polish vaccine-deniers’ comments on Facebook
Krzysztof Klimiuk
a
, Agnieszka Czoska
b
, Karolina Biernacka
a
, and Łukasz Balwicki
c
a
Faculty of Medicine, Medical University of Gdańsk, Gdańsk, Poland;
b
Sentimenti, Poznań, Poland;
c
Department of Public Health and Social Medicine,
Medical University of Gdańsk, Gdańsk, Poland
ABSTRACT
Introduction: Vaccinations are referred to as one of the greatest achievements of modern medicine.
However, their eectiveness is also constantly denied by certain groups in society. This results in an
ongoing dispute that has been gradually moving online in the last few years due to the development of
technology. Our study aimed to utilize social media to identify and analyze vaccine-deniers’ arguments
against child vaccinations.
Method: All public comments posted to a leading Polish vaccination opponents’ Facebook page posted
between 01/05/2019 and 31/07/2019 were collected and analyzed quantitatively in terms of their content
according to the modied method developed by Kata (Kata, 2010). Sentiment analysis was also
performed.
Results: Out of 18,685 comments analyzed, 4,042 contained content covered by the adopted criteria:
conspiracy theories (28.2%), misinformation and unreliable premises (19.9%), content related to the safety
and eectiveness of vaccinations (14.0%), noncompliance with civil rights (13.2%), own experience
(10.9%), morality, religion, and belief (8.5%), and alternative medicine (5.4%). There were also 1,223 pro-
vaccine comments, of which 15.2% were oensive, mocking, or non-substantive. Sentiment analysis
showed that comments without any arguments as well as those containing statements about alternative
medicine or misinformation were more positive and less angry than comments in other topic categories.
Conclusions: The large amount of content in the conspiracy theory and misinformation categories may
indicate that authors of such comments may be characterized by a lack of trust in the scientic achieve-
ments of medicine. These ndings should be adequately addressed in vaccination campaigns.
ARTICLE HISTORY
Received 9 August 2020
Revised 19 October 2020
Accepted 8 November 2020
KEYWORDS
Anti-vaccination movement;
vaccination refusal; social
media; sentiment analysis;
misinformation
1. Introduction
Vaccinations are often described as one of the greatest public
health achievements in modern medical history
1
. This is due to
vaccinations’ huge impact on the reduction in incidence of
infectious diseases
2
. Vaccines have not only improved people’s
quality of life, but have also significantly reduced the financial
burden of infectious disease
3
. Different countries have various
health policies concerning vaccinations. After the Second
World War, until 1960, vaccinations in Poland were carried
out as ad hoc campaigns responding to particular events or
perceived threats.’ In the 1960s, (Preventive Vaccination
Schedules) were introduced in Poland, divided into compul-
sory and recommended vaccinations.
4
Currently, eleven vacci-
nations are mandatory, and provided without charge in
Poland.
5,6
However, the anti-vaccination movement has been
becoming more and more prevalent both worldwide and in
Poland.
7,
During the 71st World Health Assembly, vaccine
hesitancy was identified as one of the top 10 global health
threats
8
.
Vaccines have, since their first introduction, been accom-
panied by skepticism, as well as many critical attitudes and
actions, such as refusals to vaccinate or active campaigning
against vaccinations, that could reverse the public health
achievements of vaccination
9
. Nowadays, anti-vaccine activity
is thriving offline and online, particularly on social media
10–12
.
Ogólnopolskie Stowarzyszenie Wiedzy o Szczepieniach STOP
NOP (The National Association for Knowledge on Vaccines
“STOP NOP”), which currently has over 150,000 followers on
Facebook
13,14
, is the main opinion-leader disseminating anti-
vaccine information and providing a platform for vaccine-
refusal opinions in social media. The fan page is managed by
an eponymous foundation, whose statuses set out goals includ-
ing promotion and protection of public health based on
a holistic concept of man and the world, or prevention of
adverse vaccine reactions (NOP). It works promotes optional
vaccination and the development of documents facilitating
parents’ refusal to vaccinate their children.
15
The Association
has also been responsible for public actions such as the demon-
stration in Auschwitz prisoner uniforms, comparing manda-
tory vaccinations to Nazism.
16
Nowadays, for an increasing number of people, the Internet
has become their only source for information related to health
protection and vaccinations
17
. This is a cause for concern,
because on social media every statement has the same value,
whether it comes from an expert or a person not associated
with medical science
18
. Therefore, the process of decision-
making in regards to childhood vaccination may be
CONTACT Krzysztof Klimiuk krzysztof_klimiuk@gumed.edu.pl Faculty of Medicine, Medical University of Gdańsk, Gdańsk 80-465, Poland.
HUMAN VACCINES & IMMUNOTHERAPEUTICS
https://doi.org/10.1080/21645515.2020.1850072
© 2021 The Author(s). Published with license by Taylor & Francis Group, LLC.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/),
which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.
increasingly influenced by low-quality information
19,20
.
Specific features of online communities (that lead to the crea-
tion of thought bubbles/silos) also contribute to the problem:
people gathered around the ideas of the anti-vaccine move-
ment exist in an “echo chamber,” which prevents substantive
criticism from reaching them
21
.
There is an escalating concern about anti-vaccine sentiment
in many countries,
12
which has led to vigorous developments
in vaccine hesitancy as an area of inquiry. Studies have been
carried out using a number of methodologies and with various
goals including to study the activity and connections of mem-
bers in anti-vaccine groups,
22
the most prominent Google
search results,
10
comments posted on various social media
sites
23
and many others.
Sentiment analysis is one of the more promising techniques
in the study of online communications as it provides informa-
tion on what emotions are dominant in vaccination-related
statements, and allows us to determine which approach to
providing information about vaccines and reassuring hesitant
parents will be the most cost-effective.
24
This pragmatic
approach to vaccine-related communications, which is rooted
in commercial application and treats them similarly to con-
sumer products, gives public health workers effective tools for
dealing with vaccination hesitancy, and has been used in other
studies on the same subject
11
.
The value of such studies lies in the information on which
content/themes/ideas generate the most hesitancy and negative
emotions, and also which might be most amenable to change,
allowing for more accurate and thus more effective targeting of
online information campaigns. These studies may also equip
health-service professionals to interact better with hesitant
people who could potentially be persuaded to be vaccinated.
Studies show that people do not only have doubts about the
effectiveness or safety of vaccinations, but they may also believe
in various conspiracy theories or may be convinced that alter-
native therapies can provide them with better protection
against diseases than vaccinations
21,23
. Preparing for such
arguments and addressing them, even if they are only hinted
at, may increase the effectiveness of reassurance and
persuasion.
This paper will focus on the most common outlooks in
order to map out the extent and nature of the vaccine-related
misinformation. The main objective of this study was to under-
stand the causes for vaccine hesitancy and anti-vaccination
attitudes among users active on social media. We investigated
the arguments and emotions present in comments posted on
the most popular Polish anti-vaccine Facebook page.
2. Method
2.1. Data collection
Data collection was conducted between August 1, 2019, and
September 1, 2019. Posts and comments were collected from
the most popular Facebook page advocating for the refusal of
the mandatory vaccinations in Poland (“Ogólnopolskie
Stowarzyszenie Wiedzy o Szczepieniach STOP NOP”). All
identified comments/responses published on the page between
May 1 and July 31 were archived. Given the typical patterns of
commenting, with the great majority of comments posted in
the first few days after the original post, it was assumed that
data collection during this period would not significantly
change the results due to comments added later
25
. The data
were assembled in three separate spreadsheets (one for each
month). Along with each post, the date of posting, content, and
the number of comments was recorded. The content of com-
ments and the number of reactions (without distinguishing
between “like,” “love,” “happy” or “sad”) were recorded, as
was the gender of the comment’s author based on the users’
first or ‘given’ name. User gender was recorded as male, female,
or unknown/undetermined. No other data on commenting
users were recorded. Each comment was categorized into one
argument category (Table 1). The comments were classified
based on Kata
10
, who divided anti-vaccination content into
categories. Kata’s research concerned website content, so we
adapted his classification to analyze comments, with some
categories combined to simplify interpretation. Comments
that did not match any pre-set category were assigned the
code ‘‘0”. The categorization was carried out independently
by two researchers (K.B.; K.K.). Comments posted by the
page administration were also recorded, but were not taken
into account during the content analysis. The database creation
process is shown in Figure 1.
2.2. Sentiments analysis
All the comments collected in the study were analyzed with
Sentimenti software tools for text sentiment analysis. This soft-
ware was recently developed by W3A.PL Ltd. with the help of
European Regional Development Fund. The Sentimenti emo-
tion measurement tools comprise two complementary models:
the Plutchik’s model of eight basic emotions
26
and a simple
sentiment model in which each text is evaluated on two scales:
positive-negative polarity and general arousal. Each text is
automatically evaluated on all the 11 variables (8 emotions,
arousal, positive, and negative sentiment). Our analysis con-
sidered two emotions only: happiness and anger since only
those were significant for the study.
The Sentimenti database holds responses from 20,000
unique responders who evaluated emotions associated with
different meanings of Polish lexemes and 7000 unique respon-
ders who evaluated short phrases and texts written in Polish
(for example, opinions about hotels or medical services)
27,28
.
Each participant annotated each word or utterance on 10
scales, describing the 8 basic emotions (from 0 – lack of the
emotion, to 4), as well as a positive-negative scale and an
arousal scale.
This Sentimenti database was used to create a tool for
automatic text annotation. The solution is based on the
BiLSTM neural network architecture and described in more
details in Kocoń article
27
. As reported (ibidem), the accuracy of
automatic annotation, with each emotion annotated as present
or absent in the text, was 70%. Sentimeni analysis uses numer-
ical values to determine the ‘‘emotional loading” of each piece
of text. In the measurement model, the 11 variables (eight
emotion loadings, two sentiment values, and one arousal
value) are expressed as a percentage of the emotion saturation
or intensity. This means that each emotion can be expressed in
2K. KLIMIUK ET AL.
a text with a maximum of 100% saturation, be absent (0%) or
expressed in the text with intensity between 0 and 100%.
According to the Sentimenti developers, the accuracy of the
measurement on the training data set is high: the mean error
does not exceed 8% per text and 10% per emotion.
The happiness and anger scores expressed in the comments
collected for this study were automatically calculated by the
Sentimenti tool. The 0–100% emotion intensity annotations
are thus a reasonably accurate simulation of text interpretation
by an average Polish native speaker.
2.3. Statistical analysis
Descriptive statistics were used to analyze arguments in posts
and comments. The statistical tools included with the
Sentimenti algorithm were used to test statistical significance
of gender differences in terms of the arguments and sentiment
used in the comments (using the chi-square test). The same
statistical tool was used to calculate scores to determine the
differences in emotion intensity for individual argument
categories.
3. Results
3.1. General data structure
In the study period, 18,685 comments were collected, 4,042 of
which were manually determined and classified to the cate-
gories described in Table 1. Other comments, mostly consist-
ing of brief expressions of emotional reactions including single
emoticons/emojis and some off-topic material, were not
included in the analysis of the anti-vaccine content.
Comments made by women amounted to 13,569 (72.6%) and
by men, 4,650 (24.9%). Of the remainder, 342 (1.8%) com-
ments were posted by the studied Facebook page, and 124
(0.7%) were made by users of undetermined gender.
3.2. Content analysis of Facebook post and comments
3.2.1. Conspiracy theories/search for truth (28.2%)
The “Conspiracy theories/search for truth” theme was the most
prevalent comment category. The majority of comments in this
category were assigned to the subset of “Unique theories about
purposes of vaccination” such as depopulation/genocide
(54.5% of the subcategory), but also preparing the population
to be organ donors (6.7% of the subcategory) or claiming that
national administration is controlled by a global government
(6.7% of the subcategory).
The second most frequent subcategory in this category
included comments claiming that vaccination programs were
motivated solely by profit (20.9%), even whilst being ineffective
or harmful. Politicians, pharmaceutical companies, and health-
care professionals were said to be the beneficiaries of such
businesses. Other commenters stated that to maximize profits,
some individuals responsible for distributing vaccinations have
caused patients to develop additional diseases which would
require extra treatment in the future (4.2%). There were also
statements claiming that the aware of the dangers of vaccina-
tion, but is still protecting pharmaceutical companies and
doctors who deal with vaccine distribution (5.7%).
A proportion of comments classified in this category stated
that there have been doctors and other health professionals
Table 1. Frequency of content criteria categories among all analyzed comments
(n = 4042).
Comments analysis criteria n %
Content categories
Pro-vaccination statement or deliberating with anti-vaccination
content
1,223 -
Conspiracy theories/search for truth (795) (28.2)
Unusual theories: Unique theories about purposes of
vaccination (e.g. sterilization)
178 22.4
Profit: Vaccination policies motivated by profit 166 20.9
Rebel doctors: “Enlightened” doctors break away from the
medical establishment
104 13.1
Foolish doctors: Doctors are ignorant, fearful of sanctions 86 10.8
Cover-ups: Vaccine information withheld from the public 73 9.2
Protection: Government protects doctors/manufacturers from
liability
45 5.7
Informed choices: Encouragement to make educated decisions
for oneself/one’s children
36 4.5
Collusion: Vaccine promoters benefit from illnesses caused by
vaccines
33 4.2
Privileged knowledge: Presenting information the medical
world is unaware of/rejects
33 4.2
Fear-mongering: Dangers of diseases exaggerated to frighten
parents
21 2.6
Anti-science: Biomedicine is wrong; other ways of “knowing”
(i.e. intuition, instinct)
20 2.5
Misinformation and falsehoods (560) (19.9)
Self-referencing: Links/references to anti-vaccination “experts” 240 42.9
No references: No statistics/citations provided to support claims 137 24.5
Falsehoods: Unsupported statements made 128 22.9
Misrepresentations: Sources not used truthfully, false
conclusions drawn
55 9.9
Safety and effectiveness (395) (14.0)
Underreporting: Vaccine reactions are underreported 91 23.0
Poisons: Vaccines contain poisons/toxins/contaminants 82 20.8
Idiopathic illnesses: Vaccines cause illnesses of unknown origin
(e.g. autism, SIDS)
78 19.7
Immunity: Vaccines erode immunity, create only temporary/
ineffective immunity
57 14.4
Trivial diseases: Vaccine-preventable diseases are uncommon/
not contagious/relatively mild
39 9.9
Simultaneous vaccinations: Multiple vaccines at once increase
adverse events
25 6.3
Disease decreases: Disease incidences declined without vaccines
(i.e. from improved hygiene)
23 5.8
Civil liberties (371) (13.2)
Parental rights: Civil liberties violated by taking away parental
choice
148 39.9
Totalitarianism: Vaccine mandates are excessive government
control
119 32.1
Monitoring: Vaccine programs harass parents who do not
vaccinate
104 28.0
Personal testimonies: Stories about harmed children/personal
experiences
(306) (10.9)
Morality, religion, and ideology (241) (8.5)
Immoral acts: Vaccination involves immoral acts (e.g. child
experimentation)
206 85.5
Anti-utilitarianism: Universal vaccination sacrifices a few to
benefit many
33 13.7
Religious tenets: Vaccination is against God’s will 2 0.8
Alternative medicine (151) (5.4)
Alternative treatments: Promoting treatments superior to
vaccination (e.g. homeopathy)
71 47.0
“Back to nature”: Promoting “natural” approaches (e.g. children
should get diseases naturally)
50 33.1
Products for sale: Promoting alternative products (e.g. vitamins,
essential oils)
13 8,6
Critiquing biomedicine: Established medical knowledge is
wrong (e.g. germ theory is untrue)
13 8.6
Implied debate: Suggesting debates over if vaccination is
effective/necessary
4 2.6
HUMAN VACCINES & IMMUNOTHERAPEUTICS 3
who have endorsed the various negative claims about vaccina-
tions and the consequences of their application (13.1%).
Contrary to these beliefs, there were also assertions that doctors
could not see the truth because of their ignorance, lack of
attention, fear, or lack of knowledge from sources other than
official (10.8%), as well as mentions of parents who were
intimidated by specialists and ‘‘bullied’’ into vaccinating their
children (2.6%). Numerous comments claimed that the gov-
ernments and pharmaceutical companies are aware of the
many adverse effects of vaccinations but hide them from the
public (9.2%). In contrast, there was also a belief expressed that
the medical world is unaware of that information (4.2%). Other
commenters claimed that biomedicine is (generally) “wrong,”
with the source of this belief being given as different “ways of
knowing” such as intuition or instinct (2.5%). Moreover, some
commenters encouraged the undecided and pro-vaccine users
to make “educated” decisions for their children (4.5%),
encouraging the use of sources of information presenting mis-
information or conspiracy theories.
3.2.2. Misinformation and falsehoods (19.9%)
Misinformation and falsehoods were the second most common
theme. The most frequent comments in this category were
“Links/references to anti-vaccination” experts” (42.9%) includ-
ing links to YouTube videos in 44.2% of the subcategory.
“Misrepresentation” appeared in 9.9% of the statements ana-
lyzed in this category. Claims were made that the pharmaceu-
tical companies clearly stated in the SPC that vaccinations
caused autism or SIDS. All over-interpretations of scientific
articles were also included in this category.
Comments classified as “Falsehood” (22.9%) included
claims of vaccinated people spreading diseases that they have
gained immunity to, the lack of an effective influenza vaccine,
stating that any viral vaccine destroys the intestines, etc. The
“No references” subcategory covered the 24.5% of the com-
ments in this category that did not contain references to any
statistics, scientific articles, or reliable research.
3.2.3. Safety and effectiveness (14.0%)
The themes of the comments concerned the alleged contam-
ination of vaccinations, their potential impact on health, the
meaningfulness of the vaccination program and adverse vacci-
nation reactions. The most common sub-category related to
this category was: “Vaccine reactions are underreported”
(23.0%). Examples of such comments included stating that
most parents are not aware of vaccine reactions and do not
report them, that reports sent by parents are not included in
the statistics, or that doctors do not need to report unwanted
reactions for various reasons.
A large proportion of comments also concerned potential
adverse reactions to vaccination (19.7%) directly. The most
commonly mentioned specific adverse reactions included aut-
ism or autism spectrum disorders (24.4% of the subcategory),
epilepsy (15.4% of the subcategory), and cancer (11.5% of the
subcategory). Other complaints attributed to vaccines
included: autoimmune and genetic diseases, allergies, inferti-
lity, diabetes, Sudden Infant Death Syndrome, and many more.
Questioning whether vaccines actually conferred immunity
was also fairly common (14.4%), while 6.3% of comments in
this subcategory focused on increased adverse events with
multiple vaccines administered at once.
Numerous statements claimed that vaccines were either
contaminated or toxic (20.3%), with emphasis on vaccines
containing aluminum (19.5% of subcategory), mercury (8.5%
Figure 1. Procedures in building the database of gathered comments.
4K. KLIMIUK ET AL.
of subcategory), as well as naglaze, neurotoxins, glyphosate,
thiomersal, and many other substances.
Some comments in this category pointed out decreases in
infectious disease occurring before mass immunizations
(5.8%); credit for those was given to improvements in sanita-
tion, nutrition, and decreasing poverty. Others suggested that
some diseases should not be a target for immunization because
of their mild course or low risk of complications; such as
measles (46.2% of the subcategory), chickenpox (15.4% of the
subcategory), mumps (12.8% of the subcategory), rubella
(12.8% of the subcategory), whooping cough (10.3% of the
subcategory) and others.
3.2.4. Civil liberties (13.2%)
The most common notion expressed in the “Civil liberties”
category was the idea that the parents should always have the
final say on medical procedures their children are subjected to
(39.9%). Quoting examples of countries where vaccination
programs are non-mandatory occurred in this context.
“Monitoring” (28.0%) included content in which parents of
unvaccinated children expressed how they opposed repression/
persecution, the most frequent being financial penalties (39.4%
of the Monitoring-related comments), refusal of admission to
nursery/kindergarten (30.8% of the Monitoring-related com-
ments) or withdrawal of parental rights (9.6% of the
Monitoring-related comments).
Mandatory vaccination was often associated with totalitarian-
ism (32.1%). Mentions of human rights violations, the constitu-
tion, or the government being described as “totalitarian” were
found repeatedly in the comments assigned to this category.
3.2.5. Stories about harmed children/personal experiences
(10.9%)
Stories about harmed children and personal experiences
appeared in 10.9% of the analyzed comments. Content classi-
fied in this theme included reports of problems that had
occurred as a result of vaccinations in the commenter’s perso-
nal environment. The most commonly mentioned symptoms
were related to the nervous system (27.8% of the category),
including autism, epilepsy, convulsions, increased or decreased
muscle tone and many others. Autism appeared in 6.2% of
comments in this category.
There was a significant expression of concern regarding
subjectively perceived behavioral changes (20.9% of the cate-
gory), which manifested, inter alia, as mental retardation,
increased, and decreased tearfulness or apathy. Other reports
related to respiratory symptoms (12.4% of category), skin
(12.1% of category) and digestive symptoms (8.8% of the
category).
3.2.6. Morality, religion, and ideology (8.5%)
The morality, religion, and ideology category were the least
frequently occurring content theme. Only two comments were
identified based on the religious approach. Content to do with
anti-utilitarianism was also relatively rare (13.7% of this cate-
gory). “Immoral acts” were the most common theme in this
category (85.5%), for example linking vaccines with morally
dubious actions, such as using cells of aborted fetuses in their
production, experimenting on children, or specific events
related to crime in a medical context.
3.2.7. Alternative medicine (5.4%)
This section recorded all of the anti-vaccination comments that
promoted alternative medicine. The majority of these state-
ments (47.0%) recommended treatments such as “detoxifica-
tion from heavy metals and vaccines” (36.6% of subcategory),
homeopathy (32.4% of subcategory) but also herbalism, the use
of high doses of vitamins, energy healing, and others. The latter
included promoting alternative products (8.6%) such as a set of
diets for the treatment of autism or testing for the amount of
heavy metals in the body.
Other comments focused on a “back to nature” (33.1%)
approach, with natural methods of disease prevention includ-
ing long breastfeeding, avoiding the use of medications, and
promoting the acquisition of immunity by children through
illness.
Criticisms of biomedicine were also present in 8.6% of the
comments in this category and suggested alternative methods
for strengthening immunity. Some of the comments involved
debating about alternative solutions to vaccination (2.6%).
3.2.8. Pro-vaccination content
Comments containing pro-vaccine content were also recorded,
either antagonistic to vaccine deniers or positively encouraging
vaccination (n = 1,223). Mocking/non-substantive/offensive
content made up 15.3% of this comment category.
3.3. Gender analysis of the content
As mentioned above, there were gender differences in fre-
quency of arguments by theme category. Table 2 presents
gender distribution for each argument category.
Table 2. Quantitative distribution of comments by theme and gender. The difference between male and female authors in the frequency of comment themes was
statistically significant (Chi = 215.633, p < .001). The “other” category, including page admins and the ‘‘undetermined gender’’ was not taken into account because of
the low numbers.
Arguments Female author Male author Other
No argument 10430 3347 84
Pro-vaccination statement 727 (59,4%) 476 (38,9%) 20 (1,6%)
Conspiracy theories/search for truth 548 (68,9%) 240 (30,2%) 7 (0,9%)
Misinformation and falsehoods 374 (66,8%) 182 (32,5%) 4 (0,7%)
Safety and effectiveness 298 (75,5%) 97 (24,6%)
Civil liberties 282 (76,0%) 89 (24,0%)
Personal testimonies: Stories about harmed children/personal experiences 279 (91,2%) 25 (8,2%) 2 (0,7%)
Morality, religion, and ideology 173 (71,8%) 68 (28,2%)
Alternative medicine 125 (82,8%) 26 (17.2%)
HUMAN VACCINES & IMMUNOTHERAPEUTICS 5
Moreover, their distribution varied between the three cate-
gories of gender involved in the discussion. The “other” used
only the following four types of arguments out of eight distin-
guished in the analysis: Conspiracy theories (9), Misinformation
and falsehoods (16), Personal experiences (2) and Pro-
vaccination statement (20). However, the majority of state-
ments published by this category of authors were classified as
comments lacking arguments (67).
Female authors were more likely than male authors to post
comments with no argument, while when posting comments
containing arguments, they used arguments classified as
Alternative medicine, Personal testimonies or Safety propor-
tionally more often than the male authors.
These differences are significant also when comparing the
distribution of comments containing an argument (N = 4042;
N(male) = 1203, N(female) = 2806) or without any argument
(N = 13861, N(male) = 3347, N(female) = 10430) is taken into
account (Chi = 53.246, p < .001). The “other” category was not
taken into account because of the low number of comments.
Female authors were more active in the discussions col-
lected for the study. However, in this group the proportion of
comments containing no argument was higher than in the
comments made by male authors.
We also compared the number of pro-vaccination and anti-
vaccination arguments. The difference is statistically significant
(Chi = 74.795, p < .001) with men making 39% of pro-
vaccination arguments (N(female) = 727, N(male) = 476),
and 26% of anti-vaccination arguments (N(female) = 2019,
N(male) = 727).
3.4. Sentiment analysis – happiness and anger in the
comments
Each emotion may be expressed in a text with the maximum
intensity of 100%. When it is not present in the text, its
intensity equals 0%. In the Table 3 a summary of the automatic
emotion measurements is presented: the mean, maximum and
standard deviation (SD) values for anger and happiness
expressed in comments containing no argument or comments
assigned to one of the categories described in the previous
chapter.
3.4.1. Happiness and anger by comment types
In Figure 2, mean happiness and anger values for different
categories of comments are presented. In the analyzed sample
the comments without any arguments, as well as those contain-
ing statements about alternative medicine or misinformation
and falsehoods were more positive and less angry in their
emotional content, while personal testimonies and statements
about morality and religion expressed the most anger.
Table 3. Share of individual components depending on arguments in sentiment analysis.
argument
number of
comments
anger
(mean)
anger
(maximum)
anger
(SD)
happiness
(mean)
happiness
(maximum)
happiness
(SD)
No argument 13861 31% 100% 17% 30% 100% 21%
Alternative medicine 151 33% 80% 16% 29% 82% 16%
Civil liberties 371 43% 97% 17% 19% 69% 11%
Conspiracy theories/search for truth 795 43% 98% 16% 19% 80% 11%
Misinformation and falsehoods 560 32% 100% 17% 25% 100% 15%
Morality, religion, and ideology 241 47% 99% 18% 18% 71% 10%
Personal testimonies: Stories about harmed children/
personal experiences
306 46% 85% 16% 20% 64% 11%
Pro-vaccination statement 1223 38% 100% 16% 20% 100% 11%
Safety and effectiveness 395 42% 98% 16% 19% 100% 11%
Figure 2. Mean values for anger and happiness in sentiment analysis by comment category.
6K. KLIMIUK ET AL.
When the difference between all the argument-containing and
no-argument comments was analyzed, ANOVA showed signifi-
cant difference for both anger (F = 807.161, p < .001) and happi-
ness (F = 737.458, p < .001). The comments containing an
argument tend to be more angry (mean(argument) = 40% vs
mean(no-argument) = 31%) and less happy (mean-
(argument) = 21% vs mean(no-argument) = 30%).
When all the nine categories were taken into account,
a significant effect was observed for both emotions: anger
(F = 136.2, p < .001) and happiness (F = 101.049, p < .001).
As is shown in the chart above, a cluster of more positive
comment categories (with no argument, pointing to alternative
medicine and to misinformation) and less positive ones could
be observed, varying in their anger “loadings.” Morality and
Personal Testimonies were the most angry comments in the
sample.
In the analysis of the difference between pro-vaccination
(N = 1223) and anti-vaccination (N = 2819) arguments,
ANOVA proved significant for anger (F = 20.515, p < .001)
but not for happiness (F = 0.338, p = .56). The pro-vaccination
arguments tended to be less angry than the anti-vaccination
ones.
3.4.2. Happiness and anger by gender
As could be expected from the results of comment category
frequency by gender analysis, male and female authors differed
in the intensity of both anger (F = 8.923, p = .0028) and
happiness (F = 80.761, p < .001) expressed in their comments.
Female authors expressed more happiness (mean = 29%) than
male authors (mean = 26%) and slightly less anger
(mean = 33%) compared to male authors (mean = 34%).
4. Discussion
Our study is, to our knowledge, the first analysis of anti-
vaccination comments published on Polish social media. We
have shown that conspiracy-theories and misinformation were
the most frequent themes in the published comments, and that
anger was the most frequently expressed emotion. Our results
also show that a large majority of comments made on the anti-
vaccination Facebook page were made by women overall, and
that there were proportionately more female-authored com-
ments among anti-vaccination arguments than among pro-
vaccination arguments posted to the page we collected our
data from.
The most commonly mentioned post-vaccination adverse
reactions were autism spectrum disorders and other disorders
affecting the nervous system, which confirms the results
obtained in other studies
29
. This focus on adverse effects affect-
ing the nervous system could be explained by low awareness of
the actual frequency and nature of vaccine adverse events not
only in the general public, but even in the population of
medical students.
30
Statements about adverse reactions, often
supported by anecdotal “evidence,” lack references to evi-
dence-based medicine. They are, however, highly emotionally
charged and widely available, which makes them appear more
credible and increases the strength of association between
vaccinations and adverse reactions in the society
21
. A large
number of comments concerning adverse reactions contradict
the statistics
31,32
on vaccine injury in Poland and probably
results from the fact that the group of parents reporting vaccine
injury is remarkably active, as confirmed in another study,
where the approximately 20% of postings which were dupli-
cates created an illusion of a greater number, producing the
false notion that the injuries depicted or described occurred
frequently
21
. Vaccine deniers active on the page we analyzed do
not talk about the dangers of vaccine-preventable diseases, and
they ignore the evidence from scientific studies negating the
link between vaccines and autism
12
. This suggests that one of
the approaches to combating the anti-vaccination misinforma-
tion could consist of storytelling with narratives about unvac-
cinated children who had suffered from vaccine-preventable
diseases, countering anecdotes shared on social media by anti-
vaccination posters
23
and initiating discussions on whether the
given child’s illness was causally related to vaccination or its
lack. However, raising questions about the personal motiva-
tions of the loudest vaccine deniers is not recommended.
Some individuals seem to benefit from the widespread mis-
information about vaccination and the distrust of biomedicine.
This may be explained by a more general world-view since the
opponents of vaccination are often associated with pro-natural
groups and organizations, and support alternative and com-
plementary medicine
33
. Comments in the Alternative medicine
category referred directly to the “detoxification of vaccines,”
the removal of heavy metals from the body after vaccination, or
other therapies not supported in evidence-based medicine. We
also, disturbingly, found communications offering paid thera-
pies and diets that purportedly treat autism.
Anti-vaccine groups often encourage parents to make an
“informed choice” while providing unsubstantiated and mis-
leading information
10
. Moreover, the notions of “medical lib-
erty” remain popular, with the rhetoric of anti-vaccination
activists unchanged for a few decades since the time when
they opposed vaccination legislation
34
. In Poland,
a mandatory vaccination policy is in operation
6,35
to maintain
a high vaccination rate. Although mandatory vaccination poli-
cies could be considered controversial from the individual
liberty point of view,
36
herd immunity typically requires
85–95% vaccination uptake within a population
37
, from
which the implication can be derived that vaccinating a child
is not up to the “individual decision of the parent”
21
. The
analysis of the comments we collected suggests that financial
penalties are effective in forcing opponents of vaccination to
undergo at least some vaccinations. The commenters are aware
that avoiding vaccination in a society with a mandatory vacci-
nation policy bears the mark of “free-riding,” i.e. obtaining
benefits without bearing the risk
12
. Zarobkiewicz et al. have
shown that there is a belief that non-vaccination of children
would be penalized
30
. These concerns could be strengthened
further by clear labeling of the behavior of anti-vaccine move-
ment as the aforementioned “free-riding.”
Although vaccine hesitancy may have many causes, a high
level of mistrust in conventional medicine is surely a leading
one
38
. Regrettably, it is frequently a mistrust that is directed
toward governments and pharmaceutical companies being
allegedly implicated in world-scale conspiracies and is based
on unclear premises
29
. Our study shows how much of the
discourse with vaccine-hesitant individuals is occupied by
HUMAN VACCINES & IMMUNOTHERAPEUTICS 7
such presumptions. Predominance of conspiracy arguments
and decidedly non-mainstream sources can lead to increas-
ingly extremist beliefs, and consequently to ideological isola-
tion, where the penetration of evidence-based medicine
becomes impossible and the anti-vaccine communities func-
tion in impenetrable echo chambers
39
. It also happens that
conspiracy-focused anti-vaccination attitudes are sometimes
supported by people with academic degrees
30,40
. Worryingly,
such opinions and disinformation are also sometimes repeated
and spread by prominent politicians
23
. The high frequency of
conspiracy theories concerning genocide/depopulation sug-
gests that a large number of people among the vaccine-
hesitant perceive vaccines as a threat unequivocally detrimental
to human health and seek explanation for the situation.
Studies show that religion-related content is of relatively
lesser prominence in anti-vaccination communications.
29
Our analysis shows an even lower proportion of religion-
related comments, which might be explained by the fact that,
unlike in Poland, in the US it is possible to avoid vaccination in
some states by citing philosophical/religious exception
41
.
Polish society is predominantly Catholic, and this denomina-
tion has no religious objections to the use of vaccines
38
.
The observed gender bias, with female-authored comments
dominating the collected data may indicate that women have
more concerns to do with children’s vaccination. It is reason-
able to assume that a significant proportion of these comments
come from mothers, or prospective mothers. This gender bias
was also demonstrated in other research focused on groups and
sites related to the anti-vaccination movement
22,23
.
Anti-vaccine arguments are in many cases emotionally
loaded. As indicated by our results, the comments that convey
no argument are positive in terms of emotional load and
communicate the most happiness of all the comment types
analyzed, which is similar to results obtained by Faasse et al.
11
as are our results concerning positive emotions (happiness) in
anti-vaccine arguments.
11
However, in contrast to the results
reported by Faasse et al., our analysis shows that pro-vaccine
comments were less “angry” than anti-vaccine comments. The
arguments against vaccination which proposed alternative
medicine solutions and pointed out alleged misinformation,
namely the ones that promoted “informed choice,” were also
rather positive in their emotional tone, which may also encou-
rage parents to join the anti-vaccination movement.
Researchers in another study also found that the “Food as
medicine” topic was characterized by a more positive senti-
ment than other topics,
22
similarly to the comparable “alter-
native medicine” category in our analysis. On the other hand,
the arguments which typically expressed the highest amount of
anger were the Personal testimonies and the ones concerning
Morality and religion. As a result, they may also be very
persuasive.
Users of the Internet and social media include not only
supporters and opponents of vaccinations, but also numerous
parents who are vaccine-hesitant. Despite the necessity to
develop an adequate strategy to promote vaccination to vaccine-
hesitant parents, the most effective model of persuasion has still
not been determined
42
. It was shown that previous educational
strategies were not successful in encouraging parents to vacci-
nate, and some may even have had an inverse effect
43
. However,
research conducted on a considerably larger sample of utter-
ances collected from Twitter shows that pro-vaccine content is
much more widespread than anti-vaccine posts
40
. We also know
that the discussion about vaccination suggests the existence of
two deeply polarized groups who only process information
either in favor of or against vaccines
44
. Therefore, it is crucial
for pro-vaccine users’ statements to be correctly structured.
They should not be offensive or repeat anti-vaccine rhetoric,
but they ought to improve general knowledge about
vaccinations
33
. In our study, many of the “pro-vaccine” com-
ments were shown to be mocking and abusive.
Our study has several strengths. The categorization of the
arguments in the comments was performed by human eva-
luators, which allowed us to obtain results unavailable to
a computer algorithm. This allowed for an accurate interpre-
tation of the content of the utterances. Language features,
such as sarcasm or metaphor were less likely to cause
a misunderstanding of the statement in comparison to cate-
gorization depending on algorithms
40
. Furthermore, the man-
ual coding of users’ gender was also potentially more accurate
than when coding gender automatically using a name
database
22
. Despite the process of interpretation being time-
consuming, a large number of comments were analyzed.
However, the efficiency was not comparable with what can
be achieved with the help of specialized analysis software.
Nonetheless, this is the first study in Poland attempting to
quantify the arguments in comments in the online anti-
vaccination environment. What also can be considered
a success is the fact that the study was conducted on
a Facebook page that gathers vaccine deniers, as there is
a tendency in those groups to transition into less public
arenas outside of the social media altogether in the face of
mainstream criticism
45
.
5. Limitations
Despite the advantages of the manual evaluation of the col-
lected data described above, it is worth mentioning the disad-
vantages of this approach. Some statements were difficult to
unambiguously categorize, and the boundaries between cate-
gories were not always clear. Such evaluation is necessarily
subjective, even if conducted based on well-defined criteria.
A bigger limitation was the methodology’s focus on argu-
ments without any validity, so that comments which potentially
could have some merit, such as comments criticizing the quali-
fication of children for vaccination or current medical proce-
dures were not taken into account. Manual categorization of
comments is also more time-consuming than similar tasks per-
formed by a computer algorithm and leads to a smaller sample
than in similar studies which used machine categorization.
The coding frame used to classify the comments was not
comprehensive, as demonstrated by a large proportion of com-
ments that were not assigned to any category. Moreover, pic-
tures, which are responsible for a significant part of the
emotional impact of personal narratives were not taken into
account
21
. Our study aimed to present a quantitative evalua-
tion of the content vaccine-hesitant parents may encounter
online, focusing on the relative proportion of comments
devoted to various topics. However, the relative impact of
8K. KLIMIUK ET AL.
different comments may vary and so could individual inter-
pretation by each user
29
. It is also possible that, for example,
some users only read short comments, avoiding more extensive
statements.
One of the most significant limitations of this study is that it
was limited to a single Facebook page, without a wider analysis
of Facebook or other online content. We might have also missed
some of the most extreme comments or those that questioned
the page’s motivations or operations, which may have been
removed either by Facebook or the page administrators.
A further distortion of the data might have occurred due to
the fact that one person could post the same comment multiple
times, which is a known limitation of using social media data
46
.
What’s more, no user data have been collected, so it is not
known how many comments came from each user. There may
be a group of extremely active users responsible for a large
proportion or even the majority of the content posted to the
page and it may have distorted our analysis.
For the most, sentiment analysis is based on the identification
of keywords and phrases. Such an approach is never completely
accurate, as it would require a full knowledge of the language
habits of the authors of the statements, taking into account each
specific individual word usage as well as the contextual varia-
bility of meanings. Sentiment analysis of statements formulated
in informal language is particularly difficult
46,47
. A large number
of very short statements, as is typical of Facebook comments,
also hinders the operation of the algorithm.
6. Conclusions
Although we expected to encounter a particular type of
content researching the comments on a vaccine deniers’
Facebook page, the number of comments as well as specific
content are troubling, especially the high proportion of con-
spiracy theories and misinformation-spreading comments.
The highly emotional character of anti-vaccine comments
strongly suggests that vaccine deniers, at least the ones active
in social media antivaccine spaces, hold views that would be
very difficult to change, and pro-vaccination messages that
do not address emotional connotations may prove
ineffective.
However, anti-vaccine activists make, as of now, a relatively
small if very vocal group. Being aware of their arguments,
public health institutions can prepare the rest of society for
confrontation with “anti-vaccine indoctrination.” The results
of our study suggest that the key to achieving this goal is
focusing on combating distrust by making sources of unbiased
information on vaccinations available, ensuring transparency
of procedures, and support of credible and trustworthy experts
in creating pro-vaccine content.
Article highlights
●The largest category of comments concerned conspiracy theories and
misinformation.
●Autism was the most-mentioned post-vaccination reaction.
●The pro-vaccination arguments tended to be less angry than the anti-
vaccine ones.
Acknowledgments
We want to thank Sentimenti team (https://sentimenti.com) for help in
emotional analysis of the data.
Author contributions
K.K. collected the data, drafted and drafted the manuscript, analyzed the
data, and conducted critical revisions. A.C. drafted the manuscript and
analyzed the data. K.B. collected and analyzed the data. Ł.B. conceived the
manuscript and conducted critical revisions. All authors have read and
agreed to the published version of the manuscript.
Disclosure of potential conicts of interest
There are no potential conflicts of interest to disclose.
Funding
Part of the research (sentiment analysis) was co-financed by the National
Centre for Research and Development, Poland, grant no [POIR.01.01.01-
00-0472/16] – Sentimenti (https://sentimenti.com/eu-grant/).
ORCID
Krzysztof Klimiuk http://orcid.org/0000-0002-2429-2867
Agnieszka Czoska http://orcid.org/0000-0003-1464-6147
Karolina Biernacka http://orcid.org/0000-0001-9883-1432
Łukasz Balwicki http://orcid.org/0000-0002-8821-7911
References
1. Poland GA, Jacobson RM. Understanding those who do not under-
stand: a brief review of the anti-vaccine movement. Vaccine.
2001;19(17–19):2440–45. doi:10.1016/S0264-410X(00)00469-2.
2. André FE. Vaccinology: past achievements, present roadblocks and
future promises. Vaccine. 2003;21(7–8):593–95. doi:10.1016/
S0264-410X(02)00702-8.
3. Rémy V. The economic value of vaccination: why prevention is
wealth. J Mark Access Health Policy. 2015:3. doi:10.3402/jmahp.
v3.29284.
4. Kostrzewski J, Magdzik W, Naruszewicz-Lesiuk D. Choroby
zakaźne i ich zwalczanie na ziemiach polskich w XX wieku. 1st
ed. PZWL; 2001.
5. Szczepienia obowiązkowe i zalecane - ministerstwo zdrowia - por-
tal gov.pl; [accessed: 2020 Oct 05]. https://www.gov.pl/web/zdro
wie/szczepienia-obowiazkowe-i-zalecane.
6. Minister Zdrowia. Rozporządzenie Ministra Zdrowia z dnia 18
sierpnia 2011 r. w sprawie obowiązkowych szczepień ochronnych.
Dziennik Ustaw nr 182, poz.1086.
7. Maltezou HC, Wicker S, Borg M, Heininger U, Puro V,
Theodoridou M, Poland GA. Vaccination policies for health-care
workers in acute health-care facilities in Europe. Vaccine. 2011;29
(51):9557–62. doi:10.1016/j.vaccine.2011.09.076.
8. World Health Organisation. Ten threats to global health in 2019;
2019. https://www.who.int/news-room/feature-stories/ten-threats-
to-global-health-in-2019.
9. Hodge JG, Gostin LO. School vaccination requirements: historical,
social, and legal perspectives. Kentucky Law J (Lexington, Ky).
2001;90:831–90.
10. Kata A. A postmodern Pandora’s box: anti-vaccination misinfor-
mation on the Internet. Vaccine. 2010;28(7):1709–16. doi:10.1016/
j.vaccine.2009.12.022.
11. Faasse K, Chatman CJ, Martin LR. A comparison of language use
in pro- and anti-vaccination comments in response to a high
HUMAN VACCINES & IMMUNOTHERAPEUTICS 9
profile Facebook post. Vaccine. 2016;34(47):5808–14. doi:10.1016/
j.vaccine.2016.09.029.
12. Evrony A, Caplan A. The overlooked dangers of anti-vaccination
groups’ social media presence. Hum Vaccin Immunother. 2017;13
(6):1475–76. doi:10.1080/21645515.2017.1283467.
13. Ogólnopolskie stowarzyszenie wiedzy o szczepieniach STOP NOP |
Facebook; [accessed 2020 Oct 05]. https://www.facebook.com/stow
arzyszeniestopnop/.
14. Hoffman M. Antyszczepionkowcy – antysystemowcy? Krytyczna
działalność ogólnopolskiego stowarzyszenia wiedzy o szczepieniach
STOP NOP; 2016. [accessed 2020 Apr 12]. http://dspace.uni.lodz.
pl:8080/xmlui/handle/11089/25732.
15. Statut ogólnopolskiego stowarzyszenia wiedzy o szczepieniach STOP
NOP” – ogólnopolskie stowarzyszenie wiedzy o szczepieniach “STOP
NOP”; [accessed 2020 Oct 05]. https://stopnop.com.pl/statut-
ogolnopolskiego-stowarzyszenia-wiedzy-o-szczepieniach-stop-nop/
16. Obowiązek szczepień jak nazizm? Muzeum Auschwitz oburzone -
Społeczeństwo - rp.pl; [accessed 2020 Oct 05]. https://www.rp.pl/
Spoleczenstwo/190609778-Obowiazek-szczepien-jak-nazizm-
Muzeum-Auschwitz-oburzone.html.
17. Tafuri S, Gallone MS, Cappelli MG, Martinelli D, Prato R,
Germinario C. Addressing the anti-vaccination movement and
the role of HCWs. Vaccine. 2014;32(38):4860–65. doi:10.1016/j.
vaccine.2013.11.006.
18. Gerstenfeld PB, Grant DR, Chiang C-P. Hate online: a content
analysis of extremist internet sites. Anal Social Issues Public Policy.
2003;3(1):29–44. doi:10.1111/j.1530-2415.2003.00013.x.
19. Sak G, Diviani N, Allam A, Schulz PJ. Comparing the quality of
pro- and anti-vaccination online information: a content analysis of
vaccination-related webpages. BMC Public Health. 2016;16(1):38.
doi:10.1186/s12889-016-2722-9.
20. Betsch C, Brewer NT, Brocard P, Davies P, Gaissmaier W, Haase N,
Leask J, Renkewitz F, Renner B, Reyna VF, et al. Opportunities and
challenges of Web 2.0 for vaccination decisions. Vaccine. 2012;30
(25):3727–33. doi:10.1016/j.vaccine.2012.02.025.
21. Ma J, Stahl L. A multimodal critical discourse analysis of
anti-vaccination information on Facebook. Libr Inf Sci Res.
2017;39(4):303–10. doi:10.1016/j.lisr.2017.11.005.
22. Smith N, Graham T. Mapping the anti-vaccination movement on
Facebook. Inf Commun Soc. 2019;22(9):1310–27. doi:10.1080/
1369118X.2017.1418406.
23. Hoffman BL, Felter EM, Chu KH, Shensa A, Hermann C,
Wolynn T, Williams D, Primack BA. It’s not all about autism: the
emerging landscape of anti-vaccination sentiment on Facebook.
Vaccine. 2019;37(16):2216–23. doi:10.1016/j.vaccine.2019.03.003.
24. O nas | Sentimenti; [accessed 2020 Apr 12]. https://sentimenti.pl/
o-nas/.
25. Lerman K, Ghosh R. Information contagion: an empirical study of
the spread of news on digg and Twitter social networks. ICWSM.
2010;10:90–97.
26. Plutchik R. A psychoevolutionary theory of emotions. Social Sci
Inf. 1982;21(4–5):529–53. doi:10.1177/053901882021004003.
27. Kocoń J, Zasko-Zielińska M, Miłkowski P. Multi-level analysis and
recognition of the text sentiment on the example of consumer
opinions. In: International Conference Recent Advances in
Natural Language Processing, RANLP. Vol 2019-Septe; Varna,
Bulgaria; 2019. p. 559–67. doi:10.26615/978-954-452-056-4_066.
28. Kocoń J, Janz A, Riegel M, Wierzba M, Marchewka A, Czoska A,
Grimling D, Konat B, Juszczyk K, Katarzyna K, et al. Recognition
of emotions, valence and arousal in large-scale multi-domain text
reviews. 9th Language & Technology Conference: Human
Language Technologies as a Challenge for Computer Science and
Linguistics; 2019; Poznań, Poland.
29. Moran MB, Lucas M, Everhart K, Morgan A, Prickett E. What
makes anti-vaccine websites persuasive? A content analysis of
techniques used by anti-vaccine websites to engender
anti-vaccine sentiment. J Commun Healthc. 2016;9(3):151–63.
doi:10.1080/17538068.2016.1235531.
30. Zarobkiewicz MK, Zimecka A, Zuzak T, Cieślak D, Roliński J,
Grywalska E. Vaccination among Polish university students.
Knowledge, beliefs and anti-vaccination attitudes. Hum
Vaccin Immunother. 2017;13(11):2654–58. doi:10.1080/
21645515.2017.1365994.
31. Narodowy instytut zdrowia publicznego -państwowy zakład
higieny zakład epidemiologii chorób zakaźnych i nadzoru,
główny inspektorat sanitarny departament zapobiegania oraz
zwalczania zakażeń i chorób zakaźnych u ludzi. szczepienia
ochronne w polsce w 2017 roku. Warszawa; 2018
32. Narodowy Instytut Zdrowia Publicznego. System nadzoru nad
niepożądanymi odczynami poszczepiennymi w Polsce
[Internet]. PZH2018; 2016 [accessed 2020 Apr 13]. https://
www.pzh.gov.pl/system-nadzoru-nad-niepozadanymi-odczy
nami-poszczepiennymi-w-polsce/vaccinedeniersinpublic. World
Health Organization Regional oce for Europe First Edition.
www.euro.who.int/pubrequest.
33. Marchewka AK, Majewska A, Młynarczyk G. Działalność ruchu
antyszczepionkowego, rola środków masowego komunikowania
oraz wpływ poglądów religijnych na postawę wobec szczepień
ochronnych. Postepy Mikrobiologii. 2015;54:95–102.
34. Rada Ministrów, Minister Zdrowia. Ustawa z dnia 5 grudnia 2008
r. o zapobieganiu oraz zwalczaniu zakażeń i chorób zakaźnych
u ludzi. Dziennik Ustaw nr 234, poz. 1570.
35. Hobson-West P. Understanding vaccination resistance: moving
beyond risk. Health Risk Soc. 2003;5(3):273–83. doi:10.1080/
13698570310001606978.
36. Schmid P, Rauber D, Betsch C, Lidolt G, Denker ML. Barriers of
influenza vaccination intention and behavior - A systematic review
of influenza vaccine hesitancy, 2005–2016. PLoS ONE. 2017;12(1):
e0170550. doi:10.1371/journal.pone.0170550.
37. Blankenship KL, Wegener DT. Opening the mind to close it:
considering a message in light of important values increases mes-
sage processing and later resistance to change. J Pers Soc Psychol.
2008;94(2):196–213. doi:10.1037/0022-3514.94.2.94.2.196.
38. Pospischil A. Nauczycielka akademicka podczas zajęć podważa
sens szczepień [Internet]. Wyborcza. Lublin: Uniwersytet
Medyczny w Lublinie; 2019 [accessed 2020 Apr 13]. https://
lublin.wyborcza.pl/lublin/7,48724,25295169,uniwersytet-
medyczny-w-lublinie-nauczycielka-akademicka-podczas.html?
disableRedirects=true.
39. Gunaratne K, Coomes EA, Haghbayan H. Temporal trends in
anti-vaccine discourse on Twitter. Vaccine. 2019;37(35):4867–71.
doi:10.1016/j.vaccine.2019.06.086.
40. NCSL. States with religious and philosophical exemptions from
school immunization requirements [Internet]. NCSL2020;
[accessed 2020 Apr 17]. https://www.ncsl.org/research/health/
school-immunization-exemption-state-laws.aspx.
41. Grabenstein JD. What the World’s religions teach, applied to
vaccines and immune globulins. Vaccine. 2013;31(16):2011–23.
doi:10.1016/j.vaccine.2013.02.026.
42. Nyhan B, Reifler J, Richey S, Freed GL. Effective messages in
vaccine promotion: A randomized trial. Pediatrics. 2014;133(4):
e835–e842. doi:10.1542/peds.2013-2365.
43. Hunter RF, Gough A, O’Kane N, McKeown G, Fitzpatrick A,
Walker T, McKinley M, Lee M, Kee F. Ethical issues in social
media research for public health. Am J Public Health. 2018;108
(3):343–48. doi:10.2105/AJPH.2017.304249.
44. Gass RH, Seiter JS. Persuasion : social influence and compliance
gaining. Pearson; 2013.
45. Pilkington E, Glenza J. Facebook under pressure to halt rise of
anti-vaccination groups. The Guardian; 2019 [accessed 2020 Apr
13]. p. 12–14. https://www.theguardian.com/technology/2019/feb/
12/facebook-anti-vaxxer-vaccination-groups-pressure-
misinformation.
46. Tomanek K. Analiza sentymentu - metoda analizy danych
jakościowych : przykład zastosowania oraz ewaluacja słownika RID
i metody klasyfikacji Bayesa w analizie danych jakościowychk.
Przeglad Socjologii Jakosciowej. 2014 [accessed: 2020 Apr 18];10
(2):118–36. www.przegladsocjologiijakosciowej.org.
47. Liu B. Sentiment analysis and opinion mining. Morgan & Claypool
Publishers.
10 K. KLIMIUK ET AL.
