Content uploaded by Shahan Ali Memon
Author content
All content in this area was uploaded by Shahan Ali Memon on Aug 09, 2020
Content may be subject to copyright.
Content uploaded by Shahan Ali Memon
Author content
All content in this area was uploaded by Shahan Ali Memon on Aug 01, 2020
Content may be subject to copyright.
COVID-19 Misinformation Annotation Codebook: Twitter
Shahan Ali Memon and Kathleen M. Carley
School of Computer Science
Carnegie Mellon University
5000 Forbes Ave, Pittsburgh, PA 15213, USA
{samemon,kathleen.carley}@cs.cmu.edu
1 Introduction
Detection and characterization of the different themes around COVID-19 has become a topic of interest
for many researchers. In this document, we present a comprehensive codebook related to the themes
around the Twitter COVID-19 discourse specifically in the context of misinformation. This codebook has
been created after many discussions, and we release it for the researchers as a “call to arms” to use for
collecting further annotations and data. The first iteration of the dataset collected called CMU-MisCov19,
using this codebook, along with a set of preliminary analyses can be found in [1].
2 Coding Scheme
Table 1 shows the list of categories we will consider for annotating Tweets related to COVID-19 misinforma-
tion on Twitter.
Table 1. This table describes the categories along with their IDs
ID Category
0 Irrelevant
1 Conspiracy
2 True Treatment
3 True Prevention
4 Fake Cure
5 Fake Treatment
6 False Fact or Prevention
7 Correction/Calling out
8 Sarcasm/Satire
9 True Public Health Response
10 False Public Health Response
11 Politics
12 Ambiguous/Difficult to Classify
13 Commercial Activity or Promotion
14 Emergency Response
15 News
16 Panic Buying
2 Shahan Ali Memon and Kathleen M. Carley
3 Description
The following section describes each of the categories in detail, along with their corresponding examples
and justifications.
0. Irrelevant
A tweet shall be classified as irrelevant if it may or may not mention COVID-19 or SARS-Cov-2, but if it
cannot be classified in any of the other categories below.
Example(s):
Tweet Justification
“If you’re feeling like it, today is ”Update Friday”
so dip into a channel relevant to your interests
and answer a question or pose one. Let’s get
chatty.”
This tweet should be marked as irrelevant as it
is not relevant to COVID-19 or SARS-Cov-2.
“If Taylor Swift and Avril Lavigne collaborate on
a song together it would be the cure for coron-
avirus. https://t.co/90NxHkmJcc”
This tweet mentions Coronavirus, but ultimately
is unrelated in terms of content, and hence
should be marked as irrelevant.
“The cure against Coronavirus is ’Kings
& queens’ by avamax stream it now!
https://t.co/idoU6gRLHN”
This tweet mentions Coronavirus, but ultimately
is unrelated in terms of content, and hence
should be marked as irrelevant.
“@twlldun It’s not fair I was into Covid before
anyone heard of it. Now everyone is all like
‘OMG Covid 19’ like they invented it. Where
were you when it was Covid 1-18 guys? It’s
Bleach all over again it’s not fair. Am I doing this
right?”
This tweet mentions COVID-19, but ultimately
is unrelated in terms of the content, and hence
should be marked as irrelevant.
1. Conspiracy
A tweet shall be classified as a conspiracy if it endorses a conspiracy story. Some examples of conspiracy
themes related to COVID-19 include:
1. It is a bioweapon.
2. Electromagnetic fields and the introduction of 5G wireless technologies led to COVID-19 outbreaks.
3. This was a plan from Gates Foundation to increase the Gates’ wealth.
4. It leaked from the Wuhan Labs or Wuhan Institute of Virology in China.
5. It was predicted by Dean Koontz.
Examples:
COVID-19 Misinformation Annotation Codebook: Twitter 3
Tweet Justification
“Interesting interview with Prof. Frances Boyle
re. #COVID19 is indeed a perfect #Bioweapon!
Smoking gun proof! Scary shit!”
This tweet shall be marked as a conspiracy as
it is endorsing Prof. Frances Boyle’s video on
COVID-19 being a bioweapon.
“Starting to blame US for Covid-19. This is
ridiculous. It came from wuhan. CIA has stated
in leaked documents that the wuhan laboratory
is, in fact, a “hidden” bioweapon engineering
facility. The only lab in all of China rated high
enough for handling such diseases. It probably-
”
This tweet shall be marked as a conspiracy as
it endorsing the misinformation that COVID-19
was leaked from a lab in Wuhan.
“i think both 5G and covid-19 lowering im-
mune system. considering the great suspicion
that covid-19 is actually an offensive warfare
bioweapon, they maybe designed to work to-
gether, greatly increasing coronavirus lethality.
turning 5G off might be the antidote to coron-
avirus.”
This tweet shall be marked as a conspiracy as
it is endorsing the misinformation that 5G is re-
sponsible for the COVID-19 outbreak, and that
5G weakens the immune system.
2. True Treatment
A tweet shall be classified as a true treatment if it endorses a method of treatment to ease the pain (rest
and sleep, keep warm, drink plenty of liquids, etc.), and if any of the following conditions are met:
1. The treatment has been verified by the World Health Organization (WHO) site.
2. The treatment has been verified by the Center of Disease Control and Prevention (CDC) site.
3. The treatment is supported by a peer-reviewed scientific journal that appears in Ulrich’s Global Serials
Directory as both “Active” and “Refereed/Peer-reviewed”.
4. The treatment is supported by a publicly posted working paper authored or co-authored by tenure track
faculty at a university in the top-800 universities worldwide according to the Times Higher Education
World University Rankings 2019.
5. Tweet links directly to news story which correctly cites a peer-reviewed journal article (using standards
above to adjudicate the journal).
Example:
Tweet Justification
“Mild fever itchy throat doc says sleep well drink
fluids (the non alcoholic kind he stressed). I
feel anxious about covid then remember I have
these symptoms 4 times every year.”
This tweet shall be marked as a true treatment
as the tweet endorses some of the treatments
for self care verified by WHO.
3. True Prevention
A tweet shall also be classified in this category if it explicitly endorses a method of prevention and any of
the following conditions are met:
1. The prevention has been verified by the World Health Organization (WHO) site.
2. The prevention has been verified by the Center of Disease Control and Prevention (CDC) site.
4 Shahan Ali Memon and Kathleen M. Carley
3. The prevention is supported by a peer-reviewed scientific journal that appears in Ulrich’s Global Serials
Directory as both “Active” and “Refereed/Peer-reviewed”.
4. The prevention is supported by a publicly posted working paper authored or co-authored by tenure track
faculty at a university in the top-800 universities worldwide according to the Times Higher Education
World University Rankings 2019.
Some examples of the known true preventions of the COVID-19 disease from the CDC site include:
1. Washing your hands often
2. Avoiding close contact
3. Covering your mouth and nose
4. Covering coughs and sneezes
5. Cleaning and disinfecting
6. Monitoring your health
Note: A tweet “No, cocaine does not prevent coronavirus” would not fall into this category as while it may
be preventative (i.e. preventing people from cocaine), it is not a prevention for the COVID-19 disease itself.
Examples:
Tweet Justification
“Bleach sleeping pads and masks: What the mil-
itary and Veterans Affairs are asking for to com-
bat coronavirus. . . https://t.co/ljmyUkWF0E.”
This tweet shall be marked as a true preven-
tion as it links out to a credible news source
that implicitly mentions preventative guidelines
by the CDC to assist in stopping the spread of
the virus.
“Personally my daily life really hasn’t changed
any. I believed in good hygiene before coron-
avirus and I will after. I already had an appropri-
ate stock of hand sanitizer, antibacterial wipes,
Lysol spray, bleach, body washes, and house
hold cleaners. Its called hygiene.”
This tweet shall be marked as a true preven-
tion as it encourages good hygiene which is en-
dorsed by WHO.
4. Fake Cure
A tweet shall be classified as a fake cure if the content endorses a cure and any of the following conditions
are met:
1. The cure cannot be verified by the World Health Organization (WHO) site.
2. The cure cannot be verified by the Center of Disease Control and Prevention (CDC) site.
3. The cure is not supported by a peer-reviewed scientific journal that appears in Ulrich’s Global Serials
Directory as both “Active” and “Refereed/Peer-reviewed”.
4. The cure is not supported by a publicly posted working paper authored or co-authored by tenure track
faculty at a university in the top-800 universities worldwide according to the Times Higher Education
World University Rankings 2019.
Examples:
COVID-19 Misinformation Annotation Codebook: Twitter 5
Tweet Justification
“Leaked medical conference documents reveal
US hospitals preparing for 96 million coron-
avirus infections and 480,000 deaths! ARE YOU
PREPARED? PREPARE, PREVENT, CURE
NOW: 1. With Colloidal Silver! https://keto-
longevity.com/colloidal-silver-for-longevity/. 2.
7,000 Vitamin D daily. 3. Get Masks+Goggles”
This tweet shall be marked as a fake cure as
it is advertising a cure not endorsed/verified by
WHO or CDC.
5. Fake Treatment
A treatment is different from cure as treatment improves a condition rather than completely remove the
disease. A tweet shall be classified as a fake treatment if the content endorses a treatment and any of the
following conditions are met:
1. The treatment cannot be verified by the World Health Organization (WHO) site.
2. The treatment cannot be verified by the Center of Disease Control and Prevention (CDC) site.
3. The treatment is not supported by a peer-reviewed scientific journal that appears in Ulrich’s Global
Serials Directory as both “Active” and “Refereed/Peer-reviewed”.
4. The treatment is not supported by a publicly posted working paper authored or co-authored by tenure
track faculty at a university in the top-800 universities worldwide according to the Times Higher Educa-
tion World University Rankings 2019.
Examples:
Tweet Justification
“I currently have the flu (haven’t been tested
for covid 19) and although I’m not saying that
essential oils cure or protect from it eucalyp-
tus and tea tree oil are sure helping me re-
duce a lot of my symptoms. When a vaccine
comes I’m gonna gobble that shit up though
https://t.co/PpxeRRtiue”
This tweet shall be marked as a fake treatment
as it is suggesting a treatment not endorsed by
WHO or CDC.
“How to get rid of Uneasiness in Breathing a
symptom of Covid-19. Please use Sesame oil
, Rock Salt as mentioned in Charaka Samhitha.
Ayurveda has many solutions Source: Charaka
Samhitha available Online”
This tweet shall be marked as a fake treatment
as it is advertising a treatment not endorsed by
WHO or CDC.
6. False Fact or Prevention
A tweet shall be classified as a false fact or prevention if the content mentions a false fact related to “killing”
or “disrupting” coronavirus. A tweet shall be classified as a false fact or prevention if the content implicitly or
explicitly endorses a method of prevention for coronavirus and any of the following conditions are met:
1. The prevention cannot be verified by the World Health Organization (WHO) site.
2. The prevention cannot be verified by the Center of Disease Control and Prevention (CDC) site.
3. The prevention is not supported by a peer-reviewed scientific journal that appears in Ulrich’s Global
Serials Directory as both “Active” and “Refereed/Peer-reviewed”.
4. The prevention is not supported by a publicly posted working paper authored or co-authored by tenure
track faculty at a university in the top-800 universities worldwide according to the Times Higher Educa-
tion World University Rankings 2019.
6 Shahan Ali Memon and Kathleen M. Carley
Examples:
Tweet Justification
“I heard the best way to prevent coronavirus is
to pour bleach directly into your eyes and drink
a full bottle of hand sanitizer.”
Technically this tweet is both sarcasm and fake
prevention. For the purposes of this project, this
shall be coded under false fact or prevention be-
cause: i) there are no obvious signs of sarcasm
such as an emoticon :) ; and (ii) disinformation
is often spread as anecdotes.
“2008 Research paper demonstrating var-
ious essential oil effectivity in disrupting
SARS-CoV and HSV-1 replication. #coro-
navirus #COVID-19 #COVID19 #Coron-
aVirus2020 #HSV #essentialoil #essentialoils
#sars #sarscov #sars cov #Covidcure #Hy-
droxychloroquine https://t.co/tke32spM8E
https://t.co/drzBn2nVGp”
This tweet will be marked as false fact or pre-
vention as it is directly against the WHO guid-
ance, endorses a common misinformation re-
lated to essential oils, and it tries to get credi-
bility by listing a link to a related publication.
“Garlic may help? #Covid19 Antimicrobial prop-
erties https://t.co/Anfc5SvfEy”
This tweet will be marked as false fact or pre-
vention as it is directly against the WHO guid-
ance, and it tries to get credibility by incorrectly
listing their URL.
“@CNN These things are to ‘reinforce and
boost’ inmune system. Ginger, Onions, Garlic:
anti-bacterial, anti-fungal and anti-viral proper-
ties, is known to reduce inflammation in the
body. How can you tell there is no evidence?
#coronavirus”
This tweet shall be marked as a false fact or
prevention as it is endorsing a prevention (via
boosting of immune system) which is not sup-
ported by WHO or CDC or any scientific study.
7. Correction/Calling out
A tweet shall be classified as correction if any of the following conditions are met:
1. The tweet calls out or makes fun of a fake cure, a fake prevention, fake treatment, or a conspiracy
theory.
2. The tweet links out to a site that debunks, calls out or makes fun of a fake cure, a fake prevention, fake
treatment, or a conspiracy theory.
3. The tweet calls out or make fun of violations of social distancing rules or public health responses.
4. The tweet reports/quotes a (news) story related to consequences of a false fact, fake prevention, fake
cure, fake treatment, or conspiracy theory.
5. The tweet reports/quotes a (news) story debunking a false fact, fake prevention, fake cure, fake treat-
ment, or conspiracy theory.
Examples:
COVID-19 Misinformation Annotation Codebook: Twitter 7
Tweet Justification
“Taking a hot bath, eating lots of garlic and
spraying chlorine all over your body - these
are just some of the so-called solutions to
Coronavirus that can be found on the internet.
https://t.co/CKQhmAKPRX”
This tweet can be classified as correction/calling
out as it points to a site titled: Coronavirus: So-
called ‘solutions’ debunked by World Health Or-
ganisation. The usage of the term “so-called” in
the tweet also indicates that this is a correction.
“No, #Cocaine does not protect against #coron-
avirus -French Officials.”
This tweet can be classified as a correc-
tion/calling out as it endorses the statement by
french officials calling out a specific fake preven-
tion.
“Someone told me, in support of a conspiracy
theory, ”well on the side of a bottle of bleach it
says it kills coronavirus” and I’m like...y’all there
are lots of coronaviruses that is not how this
works.”
This tweet can be classified as a correc-
tion/calling out as the author clearly de-
scribes their stance by calling out the fake
cure/prevention related to drinking bleach.
“Coronavirus myths, debunked: A cattle
vaccine, bioweapons and a $3,000 test
https://t.co/ykoLUlGspQ”
This tweet should be marked as a correc-
tion/calling out since it links out to a credible
news source debunking the claims.
“Another day another meme to debunk: Vac-
cines for the bovine coronavirus will not cure
COVID-19 https://t.co/qwHkLOnXw4”
This tweet should be marked as a correc-
tion/calling out since it links out to a credible
news source debunking the claims.
“This is what ‘the cure can’t be worse
than disease’ crowd is ok with happening.
https://t.co/0YSv1C69St.”
This tweet should be marked as a correc-
tion/calling out since it calls out a public health
response and social distancing, linking to a
credible news source.
“Coronavirus: Cocaine cure myth spreads, re-
butted by French government - Business Insider
https://t.co/rzusol40YG”
This tweet should be marked as a correc-
tion/calling out since it endorses a public health
response by the French government to debunk
misinformation.
8. Sarcasm/Satire
A tweet shall be classified as sarcasm/satire if any of the following conditions are met
1. The tweet contains clear signs of a satire calling out a fake cure, a fake prevention or a conspiracy.
2. The tweet includes a clear joke about a fake cure, a fake prevention or a conspiracy.
Concretely, this is a tweet where the information in the post is false but is presented using humor, irony,
exaggeration, or ridicule to expose and criticize people’s stupidity or vices, particularly in the context of
contemporary politics and other topical issues. This kind of post is used to ridicule other false statements or
people.
Examples:
8 Shahan Ali Memon and Kathleen M. Carley
Tweet Justification
“Which essential oil is best for getting people to
relax about the Coronavirus?”
This tweet can be classified as a satire as it is
using sarcasm to call out the essential oil fake
cures.
“Sesame oil, oregano oil and garlic. Who needs
vaccines when you can marinate?”
This tweet can be classified as a satire as it
is using sarcasm to call out the different fake
cures.
“There’s no way I can get the Coronavirus I
snorted cocaine off the back of the toilet at ala-
home I’m immune to death itself.”
This tweet can be classified as a satire as it
is using sarcasm to call out the cocaine-related
fake cure/treatment.
“Elmo isn’t scared of the Coronavirus. Elmo’s
theory is that if you do enough cocaine, it’ll kill
the virus. https://t.co/YX6gsYAVI0”
This tweet can be classified as a satire as it
is using sesame street character Elmo, and
sarcasm to call out the cocaine related fake
cure/treatment. If you follow the link in the
tweet, it also shows a clearly funny image of
Elmo snorting cocaine signalling that this tweet
should be marked as sarcasm/satire.
9. True Public Health Response
A tweet shall be classified as true public health response if it is a statement about the public health response
(eg. changes to essential services, location of testing, pending lockdown, etc.), and it links to a mainstream
news source or official government website through which the statement can be verified.
Examples:
Tweet Justification
“’Ontario and Quebec designate alcohol
producers and retailers as essential ser-
vices during current COVID-19 crisis. . .
https://t.co/EjoBZMDUy2’”
This tweet should be marked as a true pub-
lic health response as it links out to the official
Quebec government website.
“Centers for Disease Control (CDC) and Pre-
vention webpage for CORONAVIRUS (COVID-
19) https://t.co/6QueBAGPYL #KYSPIN. . .
https://t.co/M4ovNeWhNz”
This tweet should be marked as a true public
health response as it links out to the CDC web-
site on Coronavirus.
“Kudos Unilever ‘Free soap, sanitiser, bleach
and food to the value of 100 million’ ‘500
million of cash flow relief to support liveli-
hoods’ ‘We will cover our employees, contrac-
tors and others who we manage or who work on
our sites’ https://t.co/cwPsNlVoXF #coronavirus
https://t.co/sZHUR1uHU4”
This tweet should be marked as a true public
health response as the statement comes from
Unilever’s public health response for its cus-
tomers, and stakeholders.
10. False Public Health Response
A tweet shall be classified as false public health response if it makes a claim about the public health re-
sponse (eg. changes to essential services, location of testing, pending lockdown, etc.), but the claim cannot
be verified by a credible news source.
COVID-19 Misinformation Annotation Codebook: Twitter 9
Examples:
Tweet Justification
“Russia presents Covid-19 TREATMENT based
on anti-malaria drug https://t.co/D8mTTm6g3n”
This tweet should be marked as a false public
health response as it cannot be verified by the
link provided that directs to a suspicious Rus-
sian news site.
“Italy allows malaria and HIV drugs for
coronavirus treatment https://t.co/dqe91DgPv5
https://t.co/EoeURlleUX”
This tweet should be marked as a false public
health response as it cannot be verified by the
link provided that directs to a suspicious Rus-
sian news site.
11. Politics
A tweet shall be classified as politics if the tweet mentions a political individual, institution, or government
organization (eg. Congress, Democratic or Republican party), and any of the following conditions are met:
1. The tweet implicitly comments on actions taken by the political actor.
2. The tweet provides commentary on actions taken by the political actor.
Examples:
10 Shahan Ali Memon and Kathleen M. Carley
Tweet Justification
“’Trump kept saying it was basically pretty
much a cure’: Woman whose husband died af-
ter ingesting chloroquine warns the public not
to ’believe anything that the president says’
https://t.co/hWo6Zc4aOw”
This tweet should be marked as politics, since it
mentions a political actor (Trump) and is implic-
itly commenting on that actor’s statements.
“Biden on Coronavirus: ‘We Have to Take Care
of the Cure – That Will Make the Problem Worse
No Matter What‘ https://t.co/Sn0CxREPJU”
This tweet should be marked as politics, since it
mentions a political actor (Biden) and provides
commentary on his action.
“Is Nancy Pelosi serious? Covid-19 bill has
$300 million for Sesame Street & Na-
tional Endowment of Arts? An untold. . .
https://t.co/AUNx3SIEkl”
This tweet should be marked as politics, since it
mentions a political actor (Nancy Pelosi) and re-
sponds to an action taken by her (incorporated
funding for these groups in the COVID-19 bill).
“@realDonaldTrump @nytimes #Trump de-
funded & eliminated the #Pandemic Research
&; Prevention Department. #Coronavirus. . .
https://t.co/F5PZ6quYd1”
This tweet should be marked as politics, since it
mentions a political actor (Trump) and responds
to an action taken by him (defunded Pandemic
research).
“@CNN Now remember when Trump support-
ers saying the best way to get rid of the #coron-
avirus is to drink bleach.”
This tweet should be marked as politics, since it
mentions a political actor (Trump) and responds
to an action taken by his supporters.
“HAPPENING NOW: Negotiations in Congress
on #coronavirus relief & prevention.
All families regardless of immigratio. . .
https://t.co/IC80TnUj1r.”
This tweet should be marked as politics, since it
mentions a political actor (Congress) in relation
to COVID-19.
“Nailed it! ? God why aren’t Republicans praying
for a cure for .the #Coronavirus rather than sac-
rificing grandpare. . . https://t.co/njYBhofYYi.”
This tweet should be marked as politics, since
it mentions a political party (Republican) and its
supporters in relation to COVID-19.
“@Brent68189672 @rodgregg66 @theOrigi-
nalOWL @KurisuS @RudyGiuliani Trump never
said to take fish tank cleaner to remove your
COVID infection. He said the drug for malaria
treatment offers hope. If you think I’m stupid you
must also think that Cuomo is stupid for trialing
it on New Yorkers. https://t.co/6wNZiTOBM4”
This tweet can be both politics or fake treatment,
but it should be marked as politics, since the
main theme of the tweet is Trump’s statement
which is what is being disputed rather than the
malaria drug itself.
12. Ambiguous/Difficult to Classify
A tweet shall be classified as ambiguous if the stance of the author is not clear, and the post can potentially
fall into either of the contrasting categories (eg. true treatment vs. false treatment, or true prevention vs.
false prevention).
Examples:
COVID-19 Misinformation Annotation Codebook: Twitter 11
Tweet Justification
“Is it true that the tropics insulates us from covid
19? Is it true garlic is the magic bullet? Who are
most vulnerable who are less? What best first
aids? What are the numbers of first responders
to call? Where do you go to report suspected
cases? All these are not clear”
This tweet shall be classified as ambiguous as
the author is asking a question on the validity of
different fake treatments and fake preventions,
and the stance of the author is not clear.
“@nevadazmom @Meadmommy @RudyGiu-
liani Lady our entire country is shut down. There
are very good medical reasons from Medical
professionals to believe that hydroxychloroquine
or Z-pack combo could save thousands of lives
and end a looming depression that could kill mil-
lions. Why isn’t this the most important thing?”
This tweet shall be classified as ambiguous as
the stance of the author is not clear.
13. Commercial Activity or Promotion
A tweet shall be classified as commercial activity or promotion if it includes a company advertising or selling
coronavirus-related protective and preventative gear (eg. hand sanitizers, face masks, cleaners).
Examples:
Tweet Justification
“Fight Coronavirus, disinfect your home
with hypo bleach...#HypoFightCoronaVirus
.#CoronaVirusUpdate .#hypoGoWipeo
https://t.co/lMvCrQ9KIz”
This tweet shall be classified as commercial ac-
tivity or promotion as it comes from a clean-
ing supplies company and is promoting the pur-
chase of their product.
“ARE YOU IN NEED OF COVID-19 CORON-
AVIRUS CDC..SUPPLIES ? WE SELL IN BULK
SANITIZERS MASKS GLOVES..N95 BLEACH
TISSUE. . . https://t.co/k9GAI1iD5D”
This tweet shall be classified as commercial ac-
tivity or promotion as it comes from a (probable)
bot account linking to a suspicious selling web-
site.
14. Emergency Response
A tweet shall be classified in this category if it mentions a viable emergency response (eg. changes in
government funding for education programs, links to mental health resource).
Examples:
Tweet Justification
“These days can be difficult.. ?? asking
for help is brave. ..Suicide Prevention.800-
273-8255..Substance Abuse/Menta. . .
https://t.co/OiQPPjQBLL.”
This tweet should be coded under this category
as it links out to advice on mental health during
the pandemic.
12 Shahan Ali Memon and Kathleen M. Carley
15. News
A tweet shall be classified as news if it cannot be classified in any of the other categories, and it quotes a
news story and links to a news site.
Note: If the theme of the news story is about any of the above categories, it should be classified under
that category. For example, if the news story is about debunking myths related to COVID-19 or a person
dying of a fake cure, that should be classified as Correction/Calling Out. Similarly, if the news story is about
panic buying, it should be classified as panic buying.
Examples:
Tweet Justification
“‘Modern planning and civil engineering were
born out of the mid-19th century development
of sanitation in response to the spread of
malaria and cholera in cities. Digital infras-
tructure might be the sanitation of our time.’
https://t.co/561HN98RrP”
This tweet should be coded under news cate-
gory as it talks about the role of urban planning
on handling pandemics, and directly quotes the
CityLab’s article on this topic.
“10 new Utah #coronavirus cases reported
tonight—nine in Salt Lake County—bringing
our total to 19. Stay home, people!
https://t.co/Nvn4FidE0E”
This tweet should be coded under news cate-
gory as it quotes the Utah government site re-
porting the number of coronavirus cases in Salt
Lake County.
16. Panic Buying
A tweet shall be classified in this category if it mentions or comments on panic buying or its consequences
in the context of COVID-19. A tweet shall also be classified in this category if it quotes a news site/story that
talks about panic buying.
Examples:
Tweet Justification
“Found two bottles of bleach at @Target yester-
day. I’ve never been happier then finding bot-
tles of bleach before. #thelittlethings #COVID-
19 #coronavirus”
This tweet should be coded under this cate-
gory as it comments on the shortage of bleach
caused by panic buying
“No milk, no bleach: Americans awake to coro-
navirus panic buying https://t.co/LKDBkZilVg”
This tweet should be coded under this cate-
gory as it comments on the shortage of bleach
caused by panic buying
“Toilet paper, the surprise coin of the realm
of the coronavirus outbreak, was gone from
aisle 3. Most laundry detergent, bleach
and cleanser were gone from aisle 5.
https://t.co/X3znsymeEb”
This tweet should be coded under this cate-
gory as it comments on the shortage of bleach
caused by panic buying
COVID-19 Misinformation Annotation Codebook: Twitter 13
4 Additional Notes
1. If it is not clear whether a certain object is mentioned within the context of a treatment, prevention or
cure, but is essentially false, it should be classified as a false fact or prevention. For example, essential
oils have been mentioned as a “cure” to COVID-19, as well as “prevention”. If the tweet does not explicitly
mention if it is a cure or prevention, it should be classified as a “false fact or prevention”.
2. If a tweet falls into more than one category, try to infer the theme of the tweet. For example, the tweet:
“Trump never said to take fish tank cleaner to remove your COVID infection. He said the drug for malaria
treatment offers hope. If you think I’m stupid you must also think that Cuomo is stupid for trialing it on
New Yorkers.” can be classified as fake treatment or politics, but it should be marked as politics, since
the main theme of the tweet is Trump’s statement which is what is being disputed rather than the malaria
drug itself.
References
1. Memon, S.A., Carley, K.M.: Characterizing covid-19 misinformation communities using a novel twitter dataset. arXiv
preprint arXiv:2008.00791 (2020)