Content uploaded by Miguel Angel Luengo-Oroz
Author content
All content in this area was uploaded by Miguel Angel Luengo-Oroz on Sep 14, 2018
Content may be subject to copyright.
Original Paper
Crowdsourcing Malaria Parasite Quantification:An Online Game
for Analyzing Images of Infected Thick Blood Smears
Miguel Angel Luengo-Oroz1,2, PhD; Asier Arranz3, MEng; John Frean4,5, MBBCh, MMed
1Biomedical Image Technologies group, DIE, ETSI Telecomunicación, Universidad Politécnica de Madrid, CEI Moncloa UPM-UCM, Madrid, Spain
2Biomedical Research Networking Center in Bioengineering, Biomaterials and Nanomedicine, CIBER-BBN, Madrid, Spain
3Nebutek Soluciones SL, Vizcaya, Spain
4National Institute for Communicable Diseases, National Health Laboratory Service, Johannesburg, South Africa
5School of Pathology, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa
Corresponding Author:
Miguel Angel Luengo-Oroz, PhD
Biomedical Image Technologies group
DIE, ETSI Telecomunicación
Universidad Politécnica de Madrid, CEI Moncloa UPM-UCM
ETSIT, Av. Complutense 30
Madrid, 28040
Spain
Phone: 34 913366827
Fax: 34 913367323
Email: maluengo@die.upm.es
Abstract
Background: There are 600,000 new malaria cases daily worldwide. The gold standard for estimating the parasite burden and
the corresponding severity of the disease consists in manually counting the number of parasites in blood smears through a
microscope, a process that can take more than 20 minutes of an expert microscopist’s time.
Objective: This research tests the feasibility of a crowdsourced approach to malaria image analysis. In particular, we investigated
whether anonymous volunteers with no prior experience would be able to count malaria parasites in digitized images of thick
blood smears by playing a Web-based game.
Methods: The experimental system consisted of a Web-based game where online volunteers were tasked with detecting parasites
in digitized blood sample images coupled with a decision algorithm that combined the analyses from several players to produce
an improved collective detection outcome. Data were collected through the MalariaSpot website. Random images of thick blood
films containing Plasmodium falciparum at medium to low parasitemias, acquired by conventional optical microscopy, were
presented to players. In the game, players had to find and tag as many parasites as possible in 1 minute. In the event that players
found all the parasites present in the image, they were presented with a new image. In order to combine the choices of different
players into a single crowd decision, we implemented an image processing pipeline and a quorum algorithm that judged a parasite
tagged when a group of players agreed on its position.
Results: Over 1 month, anonymous players from 95 countries played more than 12,000 games and generated a database of more
than 270,000 clicks on the test images. Results revealed that combining 22 games from nonexpert players achieved a parasite
counting accuracy higher than 99%. This performance could be obtained also by combining 13 games from players trained for
1 minute. Exhaustive computations measured the parasite counting accuracy for all players as a function of the number of games
considered and the experience of the players. In addition, we propose a mathematical equation that accurately models the collective
parasite counting performance.
Conclusions: This research validates the online gaming approach for crowdsourced counting of malaria parasites in images of
thick blood films. The findings support the conclusion that nonexperts are able to rapidly learn how to identify the typical features
of malaria parasites in digitized thick blood samples and that combining the analyses of several users provides similar parasite
counting accuracy rates as those of expert microscopists. This experiment illustrates the potential of the crowdsourced gaming
approach for performing routine malaria parasite quantification, and more generally for solving biomedical image analysis
problems, with future potential for telediagnosis related to global health challenges.
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.1http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
(J Med Internet Res 2012;14(6):e167) doi:10.2196/jmir.2338
KEYWORDS
Crowdsourcing; Malaria; Image Analysis; Games for Health; Telepathology
Introduction
Crowdsourcing methodologies leveraging the contributions of
citizen scientists connected via the Internet have recently proved
to be of great value to solve certain scientific challenges
involving “big data” analysis that cannot be entirely automated
[1]. In the GalaxyZoo project, citizen scientists classified
imagery of hundreds of thousands of galaxies drawn from the
Sloan Digital Sky Survey and the Hubble Space Telescope
archive [2]. Crowdsourced contributions can be achieved with
different motivation strategies, such as micropayments or games.
The “serious games” concept refers to an intention not only to
entertain users, but also to train or educate them [3]. The
“gamification” [4] of the crowdsourcing approach enables a
higher motivation of the participants and, using the Internet as
a vehicle, untaps an underexploited resource for scientific
research [5,6]: it is estimated that 3 billion hours per week are
spent playing computer and videogames worldwide [7]. For
instance, Fold-It, an online game where players solve
3-dimensional puzzles by folding protein structures, has resulted
in several breakthrough scientific discoveries [8-10]. Another
recent growing trend is the use of crowdsourcing techniques
for participatory health research studies in which individuals
report in real time a variety of health conditions [11], providing
a promising complement to traditional clinical trials. Considering
crowdsourced image analysis, collective processing has been
recently explored for earthquake damage assessment from
remote sensing imagery [12]. However, this methodology has
not yet been mainstreamed for biomedical image analysis.
In this context, analysis of microscopic images of
malaria-infected blood samples is an appealing goal. Worldwide,
there are more than 200 million malaria cases and approximately
800,000 deaths annually, mainly in children [13,14]. Careful
optical microscopic examination of a well-stained blood film
remains the gold standard for malaria diagnosis [15].
Confirmation of a negative diagnosis is ultimately dependent
on the technician’s expertise and can take up to 20 minutes. In
addition, as malaria prevalence decreases in one specific place
over time, microscopy technician skills may now be needed in
other regions. Fast, cheap, ubiquitous, and accurate diagnosis
is a priority in the Agenda for Malaria Eradication [16].
Although automated processing methodologies have been used
extensively for the analysis of digitized blood smears [17,18],
currently there are no completely automated image processing
systems that can achieve perfect parasite recognition [19-24].
The main problem in computer-aided malaria diagnosis is that
algorithms are usually not very robust with respect to the
variable appearance of the parasites and changing image
acquisition conditions.
The goal of this research was to test the feasibility of a
crowdcomputing approach for malaria parasite quantification
in which nonexperts count parasites in digitized thick blood
smears through an online game (crowdsourcing) and a decision
algorithm combines the data generated by several players in
order to achieve a collective detection with a higher accuracy
rate than an individual analysis. This idea—gaming for
distributed malaria image analysis—has been also explored in
a recent study by Mavandadi et al [25], in parallel to and
independently of this study. These researchers designed a video
game and a processing pipeline to investigate whether
nonexperts can assess if a single-cell image extracted from a
digitized thin blood sample is infected with malaria or not.
Although this study and the present research share a similar
vision and goal, the research questions posed and solutions
adopted differ substantially in terms of the data analyzed, the
nature of the participants, the main task required of them, and
the processing methodologies.
The proposed system in this study provides a new tool for
parasite counting, but not malaria diagnosis, which is a more
complex problem [26]. For this purpose, the microscopist
protocol will need to be translated completely into a gaming
protocol, including assessing the presence or absence of
parasites, the parasite species, and growth stages and prognostic
markers, such as schizonts or gametocytes, or pigment load. In
the long run, crowdsourced remote telediagnosis from images
acquired with optical microscopy and distributed worldwide
through the Internet and possibly with systems that integrate
the microscope into mobile phones [27,28], might have a
potential impact for malaria-endemic countries because
diagnosis availability and its cost could be optimized. However,
in addition to the need for conventional laboratory processing
and imaging equipment to prepare the material to a sufficiently
high standard, this kind of analysis will require a
communications infrastructure with enough bandwidth to
distribute the images over the Internet and a critical number of
online participants in order to ensure timely analysis of the
images.
This work presents a proof-of-concept system that explores the
feasibility of an online game-based, crowdsourced solution for
malaria parasite quantitation in digitized images of thick blood
smears.
Methods
We selected an image database of malaria-positive blood films
that had been previously analyzed by experts to generate gold
standards. These images were then incorporated into an online
game. The player’s task was to click on the parasites. When a
player found all the parasites present in 1 image (constituting
a level) within a limited amount of time, the game continued
by presenting a new image. Otherwise, the game was over. All
the players’ clicks were registered in a database. After 1 month,
all the collected data was preprocessed in order to group all the
clicks that players placed around the different objects in the
image: parasites, white blood cells (leukocytes), and background
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.2http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
noise. Finally, an algorithm that combined the different games
to increase accuracy was developed and evaluated.
Ethics Statement
The malaria images used in this research were previously used
to evaluate automated image analysis methods [20]. Original
blood samples and resultant test images were collected and used
with ethical approval from the Human Research Ethics
Committee (Medical), University of the Witwatersrand,
Johannesburg, South Africa (protocol number M051126). No
new ethical review board approval was required since the digital
images used in our work were not linked to any patient data or
diagnosis and were digitally shared for microscopic training
evaluation purposes. The data analyzed in this research were
anonymously produced by online volunteers who agreed to play
an Internet game. The participants were informed of the research
purposes of the game on the game webpage.
Image Database
The image database was compiled from 28 Giemsa-stained thick
films made from blood infected with malaria (Plasmodium
falciparum) parasites, acquired using a 50× objective in a
conventional laboratory optical microscope. Medium to low
parasitemia images were selected for the game because of its
design (1-minute games) and the fact that discrepancies between
automatic counting methodologies and manual expert counting
tend to be greater in low parasitemia cases. A gold standard
mask image was generated for each of the 28 images to evaluate
player performance.
Game Architecture
The objective of the MalariaSpot game was to tag as many
parasites as possible in an image in 1 minute. The
instructions—what is a parasite and what it is not—were briefly
explained in the splash screen of the game website (Figure 1a).
During the game, if the player found all the parasites in 1 image
in the allowed time, a new image was presented (Figure 1b).
Therefore, a player could analyze several images (levels) in a
single game. In order to reinforce the game’s addictive nature,
the players were given continuous feedback: each click was
compared with the gold standard and an icon was placed
immediately at the tag position to indicate a correct or incorrect
selection. In addition, if the player misidentified an object and
clicked in a wrong location (eg, on a leukocyte), the player was
penalized by reducing the remaining time available to solve the
level. Players were confronted with different, randomly selected
test images. The difficulty of the levels increased as the time
penalty for wrong tags grew with each level. As a motivation
strategy, at the end of the game players were invited to register
and provide their name, email address, and country in order to
be included in the table of high scorers depicting the top daily,
weekly, and monthly players.
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.3http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Figure 1. MalariaSpot example screens and player distribution. (a) Splash screen of MalariaSpot game website showing the game instructions. (b)
Example of MalariaSpot game screen. (c) Map showing the geographic distribution of players during the evaluation period (the darkness of the country
is proportional to the number of visits).
Data Collection
The MalariaSpot game webpage was launched on April 25,
2012 (World Malaria Day). During the following month, more
than 6000 players from 95 different countries (Figure 1c) visited
the game webpage according to the number of Internet Protocol
(IP) addresses reported by Google Analytics, although the actual
number of players was probably larger because those connecting
from big institutions, such as universities, share the same IP
address and other players may have blocked the Google
Analytics script. Online volunteers played a total of 12,105
games that resulted in the analyses of 20,049 images and
generated a database of 270,207 tags. Social media was the
main traffic source; approximately 30% of the players originated
from a Facebook link and 30% came from a Twitter reference.
Most of the remaining visits were through links in digital
newspapers and blogs, especially from Spanish-speaking
countries.
Data Preprocessing
All the players’ clicks were saved into a database containing
the user identification number, image identification number,
x-position and y-position on the game screen, time of the click
(from the start of the level), and whether the click was on a true
parasite or not (see Figures 2a-d and Multimedia Appendix 1).
In a preprocessing step, we generated a binary matrix, In(g,p),
for each test image n, where each row gcontains a different
game and each column pcorresponds to a parasite. A value of
1 at a certain position, In(gi,pj) = 1, means that the parasite with
index jhas been clicked in the game i. Otherwise, In(gi,pj) = 0.
The number of rows is the number of games that have been
played at each test image n. The number of columns corresponds
to the number of parasites for a given level in the gold standard
plus the number of phantom parasites. We defined a phantom
parasite as an object in the image that is not a parasite and that
has been tagged by ≥ 1 players. The phantom parasites were
defined in order to group together all the clicks that were around
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.4http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
the same position, but not on the identical pixel (eg, all the clicks
that were inside a leukocyte were considered to be pointing at
the same phantom parasite). An image processing pipeline that
grouped together clicks that were at a distance of less than the
typical parasite size was implemented in order to generate all
the connected components in each image that corresponded to
phantom parasites (Figure 2c). Therefore, the output of this
preprocessing stage consisted of 1 binary matrix per test image
that characterized the performance of all the games played for
each image. Additionally, filtered versions of these matrices
were created by selecting only the data from games in which at
least 1 level was completed, 2 levels were completed, and so
on.
Figure 2. Crowdsourced image analysis of thick blood film infected with malaria. (a) Test image analyzed in the game. (b) Gold standard image in
which each label corresponds to a parasite. (c) Aggregation of gamer’s clicks where green regions correspond to correctly tagged parasites and red
regions to players’ mistakes. (d) Gamers’ clicks superimposed on raw image.
Collective Parasite Detection: Quorum Algorithm
A critical aim of this research was to show how individual
nonexpert analysis can be combined to achieve higher accuracy
rates. In order to combine the games of several players and
produce a single “detection,” we implemented a quorum
algorithm. The output of the quorum collective detection features
all the image objects (both true parasites and phantom parasites)
in one image that have been tagged in at least Xindividual games
out of a larger group of Ygames (Figure 3). The idea is simple:
an object is considered in the collective detection if it has been
tagged (“voted”) in at least X out of Y (X ≤ Y) games. Typically,
when the quorum value increases, there are fewer true positives
and false negatives. In order to evaluate the performance of
different group sizes and quorum values, we randomly selected
1000 subsets of games per (X,Y) couple with a maximum group
size of Y = 30 games. For each individual subset of Y random
games in 1 image, the collective detection performance was
measured for all quorum values of X ≤ Y. Performance
evaluation was also measured taking the subset of games that
passed at least level 1, level 2, level 3, and level 4.
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.5http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Figure 3. Illustrative example of the quorum algorithm. Parasite detection results on test image obtained from the combination of 4 games processed
with different quorum values.
Results
Out of a total of 270,207 clicks, 78.65% tagged a true parasite.
Analysis of the levels reached by the players reveals that
approximately one-third of the players were able to find all the
parasites in an image, independent of the level (see Figure 4d).
Additionally, once players successfully completed level 5, they
became game experts and no longer followed the 1 in 3 chance
of passing to the next level—they could complete as many as
22 levels (achieved by the best player so far). Interestingly, the
overall number of clicks on each of the parasites in 1 image was
similar, meaning that although 2 of 3 players usually did not
complete the level, all the parasites were equally difficult to
identify (Figures 4a and b). This fact was corroborated in a
special case for image ID6, where the probability of tagging
one particular phantom parasite was as high as the typical
probability for a true parasite. A further look into the gold
standard revealed that, in fact, this phantom parasite was a true
parasite that was not included in the gold standard by mistake
(Figure 4c).
We performed an exhaustive evaluation of the collective gamers
performance using the quorum algorithm evaluated 1000 times
for all group sizes ranging from 1 to 30 games over each of the
test images under the different training conditions (completing
1 level can be considered as a 1-minute training) (Figure 5a).
Results show a monotone smooth behavior for the true positive
(TP) and false positive (FP) rates depending on the group size
and quorum value: the bigger the group size or the smaller the
quorum required, the more true parasites were tagged and the
higher the TP rate, but also more phantom parasites were
collectively tagged, increasing the FP rate (Figure 5b). Analysis
of the discrimination index (DI) function (DI = TP – FP)
revealed that there was an optimal quorum number that
maximized the DI for each group size (Figure 5c). For instance,
the optimal quorum value was 3 for a group size of 7 games
(randomly chosen among all games) achieving a DI = 90%,
whereas the optimal quorum for a group of 10 games was 4,
providing a mean DI = 95%. When comparing the performance
of the collective analysis based on the training time (levels
completed), we observed a clear dependence between training
and DI (Figure 5d). The number of games needed to be
combined in order to achieve a DI = 99% was 22, 13, 10, 9, and
4, respectively, for the subset of games that successfully
completed 0, 1, 2, 3, and 4 game levels.
The maximum DI for each group size (obtained with its optimal
quorum value) at all training levels was fitted to a model
equation DI = f(group size, training time) using the scientific
data mining software Eureqa [29]. A multivariate optimization
process was used to find the following collective detection
equation:
DI(group,training) = 1 - e–(alpha + beta group + gamma group training)
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.6http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
where group size ranged from 1 to 30 games and the coefficient
of determination (R2) goodness of fit with (alpha = 0.69,
beta = 0.24, and gamma = 0.13) is greater than 0.97 for each
training value {0,1,2,3,4} levels (or minutes) (Figure 6). This
equation highlights the product group⋅training, meaning that
the accuracy increase provided by adding 1 game to the group
size can almost be compensated (the term beta⋅group varies
when increasing the group size) by 1 minute training or vice
versa.
We also evaluated the collective performance detection against
the automated image recognition methodology presented by
Frean [20]. For each image, we calculated the minimum number
of gamers needed to perform as well as the automatic system
and we found that it was required to combine 7.2, 4.6, 3.9, 3.0,
and 2.3 games, respectively, from the subset of players that
successfully completed 0, 1, 2, 3, and 4 game levels (see
Multimedia Appendix 2).
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.7http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Figure 4. Individual gamer’s performance. (a) Tagging probability for true parasites and phantom parasites on image ID1 based on all games. (b)
Tagging probability field superimposed on raw image. True parasites (gold standard) are signaled by white squares. (c) Aggregated tagging probability
for true parasites and phantom parasites on image ID6. Note that the probability of the phantom parasite (tagged in yellow) is as high as the true parasites.
Detailed analysis showed that it was a mistake on the gold standard. (d) Number of games played at each level.
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.8http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Figure 5. Collective parasite detection. (a) Accuracy results in 1000 random groups run over image ID1 with a group of 10 and 15 games and a quorum
of 3 and 6 votes. (b) Mean results of true positives (TP), false positives (FP), and TP – FP for 1000 experiments of all the group sizes and quorum values
with different experience. (c) Quorum values that maximize the TP – FP rate for all group sizes and training levels. (d) Maximum recognition score for
each group size and training level. Values represent the mean and standard deviation among the regular test images.
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.9http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Figure 6. Model of the collective detection of malaria parasites. Curve fitting of the mean accuracy rate for all group sizes and training levels.
Discussion
In this study, a crowdcomputing image analysis system was
developed that identifies malaria parasites in digitized images
of thick blood smears using an online game played by nonexpert
volunteers. Results for the images analyzed showed that the
performance of the quorum algorithm that combines the games
from different players can be as high as both human expert
counting and automated processing methodologies. Indeed, the
results showed that, on average, the combination of 22 games
or more, regardless of the players’ experience, was enough to
obtain almost perfect parasite counting (99%) in the tested
images. This performance could also be obtained by combining
13 games from players trained for 1 minute. However, no
conclusions could be drawn about detecting extremely low
parasitemias (as low as 1 parasite per 30 or more images).
Feedback from several players stressed the need for clearer
game instructions. Although our strategy was learning by doing,
commenters suggested that an explanation screen and/or an
initial training level would lead to better results, at least for
those players who did not complete the first level. The collective
detection equation allowed us to model the system performance
in terms of the number of games and the training of the players.
An important question arising from this model is whether any
crowdsourced image analysis system that roughly consists of
detecting spots in images will have a similar behavior. In the
affirmative case, the model could be used in the future to design
and evaluate new crowdsourced biomedical image analysis
applications.
The overall results endorse the online gaming approach to the
task of counting malaria parasites in thick blood films using a
crowdsourcing methodology, validated “in the wild” by
thousands of anonymous online players. This conclusion extends
the findings of Mavandadi et al [25], but from a different
perspective. The methodology of the present research involves
finding parasites in images from thick blood samples whereas
in the study by Mavandadi et al the main task was to make
binary decisions (infected versus uninfected) of single-cell
images extracted from thin blood samples [25]. Both thin and
thick blood films are used in malaria microscopy [26]. The thick
film, consisting of many layers of red and white blood cells, is
used to search for malaria parasites and to count them as an
indicator of the severity of the disease. The thin film, a single
layer of red and white blood cells, is mainly used to confirm
the malaria parasite species and sometimes to enumerate
parasites and evaluate other prognostic features. In addition to
the type of data analyzed (thin versus thick blood film) and the
task required of the participants (binary decisions in single-cell
images versus parasite detection and location), the nature and
number of participants varies, from 31 controlled volunteers
[25] versus > 6000 anonymous online contributors in the current
research. The studies can be considered complementary and
directed toward the same goal; the different methodologies and
experiments lead to the global conclusion that nonexperts are
able to rapidly learn and identify the typical features of malaria
parasites in digitized thin and thick blood films, and that the
combination of the analyses of several users can provide similar
accuracy rates for parasite quantification as expert microscopists.
Future developments of the current research should include the
exploration of new algorithms that combine the games of several
players in a more complex way than the quorum algorithm. For
instance, if players’ identities are logged and tracked, it would
be possible to adapt algorithms to differentially weight the
analysis of players depending on their profile, experience, past
performance, or gaming strategies [30-33]. Automated
processing methodologies report accuracy rates that are high,
but still not as good as human visual inspection; therefore, we
expect that combined man-machine diagnosis systems will be
the most effective strategy. Note that, for instance, in the present
study the players’ detection performance was similar for all the
images, whereas the automated detection algorithm [20] had
heterogeneous performances for different images. Therefore,
in hybrid systems, humans could be used first to train the
recognition algorithm and later to analyze the more complex
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.10http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
cases (supplementing the automatic processing methodologies),
whereas the easy cases would be automatically processed.
In summary, this proof-of-concept research has shown that
malaria image analysis for parasite quantification, obtained by
combining the detection of several online nonexperts with
minimal training, can be as good as the results provided by an
expert microscopist. Although the game score is generated by
comparing the user tags with previously analyzed images, in
future, the observation protocol from expert microscopists could
theoretically be translated into a game and images that have not
yet been assessed by professionals could be introduced into that
game. This raises the possibility of establishing a global
specialized task force of remote gamers-workers able to perform
online malaria parasite detection and quantitation. The validity
of this approach for malaria diagnosis is still unclear and will
depend on the method’s speed, efficiency, robustness, cost, and
above all, accuracy. Constraints related to production of the
high-quality images required for malaria species identification
will have to be addressed. Specifically, the performance should
be compared to current diagnostic tools and trends, such as rapid
immunochromatographic diagnostic tests that offer a
cost-efficient solution. However, rapid tests have limitations,
such as restricted malaria species recognition and an inability
to quantitate parasite load and monitor parasitological response
to treatment. In general, we suggest that the methodology
presented in this research could be applied to other biomedical
image analysis tasks with potential impact on global health
challenges, such as enumeration of acid-fast bacilli in sputum
smears for tuberculosis diagnosis. An inherent benefit from this
distributed telediagnosis system is that it is scalable and resilient.
Among other positive externalities of this research, there is a
clear educational impact because more than 6000 players have
learned how malaria parasites appear in thick blood films. In
addition, as we allowed players to introduce their nicknames
into the table of high scorers, we could identify approximately
100 players who now can be considered as experts in parasite
counting, within the system’s limitations. Citizen science
projects of this kind could impact future educational paradigms:
they are a clear opportunity for engaging with young people
and offer a hands-on experience that could be used in online
learning platforms [34,35].
Concerning the evolution of the MalariaSpot platform, next
steps might explore the feasibility of developing a new game
version that mimics, if possible, all the relevant steps of the
microscopist protocol in real-life conditions (eg, decisions about
presence or absence of malaria parasites, parasite stages and
species, and quantitation), but this is a much more complex and
challenging process. Assuming image quality concerns can be
addressed, this system could potentially be completed by
integrating the online platform for rapid diagnosis with the
recently developed cellphone-microscope systems [36] that
allow data transfer directly from field workers and health
centers, distributing the data worldwide through the Internet.
Acknowledgments
This research was partially funded by the Picata program from the Moncloa Campus of International Excellence, Universidad
Politécnica de Madrid and Universidad Complutense de Madrid, Spain; and the project TEC2010-21619-C04-03 from the Spanish
Ministry of Science and Innovation.
The authors would like to thank Enrique Mendoza for setting up the Web database and Jacobo Gomez for the Web illustrations.
Thanks for the support in the launch of the game to Anoush Tatevossian, Natalia Rodriguez, Maria Fernandez, Maria Luengo-Oroz,
Antonio Blanco, Jose L Rubio-Guivernau, Nati Luengo-Oroz, and Cesar Martin. Thanks for comments and feedback on the
manuscript to Carlos Castro-Gonzalez, David Pastor-Escuredo, Gert Wollny, Juan Ortuño, Pedro Guerra, Patricia Arroba, Thierry
Savy, Ana Tajadura-Jimenez, Maria J Ledesma-Carbayo, Andres Santos, and Jonathan Platkiewicz.
Last, but not least, thanks to the more than 6000 anonymous participants who played the MalariaSpot game.
Authors' Contributions
Miguel A Luengo-Oroz conceived and directed this research, designed the experiment and the MalariaSpot game, performed data
analysis, interpreted the results, and wrote the manuscript. Asier Arranz implemented the game software. John Frean provided
the image data, advised on microscopic aspects of malaria diagnosis and parasite enumeration, and contributed to revising and
editing the manuscript.
Conflicts of Interest
None declared.
Multimedia Appendix 1
Raw data collected during the experiment: Data_MalariaSpot.csv containing 270207 rows - 1 per parasite tag- in the format: [user
id, image id, true/phantom parasite, x position, y position, time].
[CSV File, 7MB - jmir_v14i6e167_app1.csv ]
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.11http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Multimedia Appendix 2
Comparison between crowdsourced parasite counting and automatic image analysis counting methodology.
[PDF File (Adobe PDF File), 17KB - jmir_v14i6e167_app2.pdf ]
References
1. Thompson K. All hands on deck. Sci Am 2012 Feb;306(2):56-59. [Medline: 22295679]
2. Lintott C, Schawinski K, Slosar A, Land K, Bamford S. Galaxy zoo: morphologies derived from visual inspection of galaxies
from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society 2008;389:1179-1189. [doi:
10.4401/ag-5324]
3. Abt C. Serious Games. Lanham, MD: University Press of America; 1987.
4. Deterding S, Dixon D, Khaled R, Nacke L. From game design elements to gamefulness: defining gamification. 2011 Sep
28 Presented at: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media
Environments; 2011; Tampere, Finland p. 9-15. [doi: 10.1145/2181037.2181040]
5. Good BM, Su AI. Games with a scientific purpose. Genome Biol 2011;12(12):135. [doi: 10.1186/gb-2011-12-12-135]
[Medline: 22204700]
6. Nielsen M. Reinventing Discovery: The New Era of Networked Science. Princeton, NJ: Princeton University Press; 2011.
7. McGonigal J. Reality Is Broken: Why Games Make Us Better and How They Can Change the World. New York: Penguin
(Non-Classics); 2011.
8. Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, et al. Predicting protein structures with a multiplayer online
game. Nature 2010 Aug 5;466(7307):756-760 [FREE Full text] [doi: 10.1038/nature09304] [Medline: 20686574]
9. Eiben CB, Siegel JB, Bale JB, Cooper S, Khatib F, Shen BW, et al. Increased Diels-Alderase activity through backbone
remodeling guided by Foldit players. Nat Biotechnol 2012 Feb;30(2):190-192. [doi: 10.1038/nbt.2109] [Medline: 22267011]
10. Lessl M, Bryans JS, Richards D, Asadullah K. Crowd sourcing in drug discovery. Nat Rev Drug Discov 2011
Apr;10(4):241-242. [doi: 10.1038/nrd3412] [Medline: 21455221]
11. Swan M. Crowdsourced health research studies: an important emerging complement to clinical trials in the public health
research ecosystem. J Med Internet Res 2012;14(2):e46 [FREE Full text] [doi: 10.2196/jmir.1988] [Medline: 22397809]
12. Barrington L, Ghosh S, Greene M, Har-Noy S, Berger J. Crowdsourcing earthquake damage assessment using remote
sensing imagery. Annals of Geophysics 2012;54(6). [doi: 10.4401/ag-5324]
13. WHO Global Malaria Programme. World Malaria Report 2011. Geneva, Switzerland: World Health Organization; 2011.
URL: http://www.who.int/malaria/world_malaria_report_2011/9789241564403_eng.pdf [accessed 2012-11-27] [WebCite
Cache ID 6CUNPBCwf]
14. Murray CJ, Rosenfeld LC, Lim SS, Andrews KG, Foreman KJ, Haring D, et al. Global malaria mortality between 1980
and 2010: a systematic analysis. Lancet 2012 Feb 4;379(9814):413-431. [doi: 10.1016/S0140-6736(12)60034-8] [Medline:
22305225]
15. Shute G. The microscopic diagnosis of malaria. In: Wernsdorfer WH, McGregor I, editors. Malaria: Principles and Practice
of Malariology. Edinburgh: Churchill Livingstone; 1988:781-814.
16. malERA Consultative Group on Diagnoses and Diagnostics. A research agenda for malaria eradication: diagnoses and
diagnostics. PLoS Med 2011;8(1):e1000396 [FREE Full text] [doi: 10.1371/journal.pmed.1000396] [Medline: 21311583]
17. Angulo J, Flandrin G. Automated detection of working area of peripheral blood smears using mathematical morphology.
Anal Cell Pathol 2003;25(1):37-49. [Medline: 12590176]
18. Luengo-Oroz M, Angulo J, Flandrin G, Klossa J. Mathematical morphology in polar-logarithmic coordinates application
to erythrocyte shape analysis. In: Marques JS, editor. Pattern Recognition and Image Analysis: Second Iberian Conference,
IbPRIA 2005, Estoril, Portugal, June 7-9, 2005, Proceedings, Part 1 (Lecture Notes in ... Pattern Recognition, and Graphics)
(Pt. 1). Berlin: Springer Berlin Heidelberg; 2005:199-206. [doi: 10.1007/11492542_25]
19. Prescott WR, Jordan RG, Grobusch MP, Chinchilli VM, Kleinschmidt I, Borovsky J, et al. Performance of a malaria
microscopy image analysis slide reading device. Malar J 2012;11:155 [FREE Full text] [doi: 10.1186/1475-2875-11-155]
[Medline: 22559294]
20. Frean JA. Reliable enumeration of malaria parasites in thick blood films using digital image analysis. Malar J 2009;8:218
[FREE Full text] [doi: 10.1186/1475-2875-8-218] [Medline: 19775454]
21. Purwar Y, Shah SL, Clarke G, Almugairi A, Muehlenbachs A. Automated and unsupervised detection of malarial parasites
in microscopic images. Malar J 2011;10:364 [FREE Full text] [doi: 10.1186/1475-2875-10-364] [Medline: 22165867]
22. Sio SW, Sun W, Kumar S, Bin WZ, Tan SS, Ong SH, et al. MalariaCount: an image analysis-based program for the accurate
determination of parasitemia. J Microbiol Methods 2007 Jan;68(1):11-18. [doi: 10.1016/j.mimet.2006.05.017] [Medline:
16837087]
23. Tek FB, Dempster AG, Kale I. Computer vision for microscopy diagnosis of malaria. Malar J 2009;8:153 [FREE Full text]
[doi: 10.1186/1475-2875-8-153] [Medline: 19594927]
24. Tek F, Dempster A, Kale I. Parasite detection and identification for automated thin blood film malaria diagnosis. Computer
Vision and Image Understanding 2010;114:21-32. [doi: 10.1016/j.cviu.2009.08.003]
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.12http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
25. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Distributed medical image analysis and diagnosis
through crowd-sourced games: a malaria case study. PLoS One 2012;7(5):e37245 [FREE Full text] [doi:
10.1371/journal.pone.0037245] [Medline: 22606353]
26. World Health Organization. Basic Malaria Microscopy: Part I. Learner's Guide. Geneva: World Health Organization; 2010.
27. Miller AR, Davis GL, Oden ZM, Razavi MR, Fateh A, Ghazanfari M, et al. Portable, battery-operated, low-cost, bright
field and fluorescence microscope. PLoS One 2010;5(8):e11890 [FREE Full text] [doi: 10.1371/journal.pone.0011890]
[Medline: 20694194]
28. Breslauer DN, Maamari RN, Switz NA, Lam WA, Fletcher DA. Mobile phone based clinical microscopy for global health
applications. PLoS One 2009;4(7):e6320 [FREE Full text] [doi: 10.1371/journal.pone.0006320] [Medline: 19623251]
29. Schmidt M, Lipson H. Distilling free-form natural laws from experimental data. Science 2009 Apr 3;324(5923):81-85
[FREE Full text] [doi: 10.1126/science.1165893] [Medline: 19342586]
30. Barrington L, Turnbull D, Lanckriet G. Game-powered machine learning. Proc Natl Acad Sci U S A 2012 Apr
24;109(17):6411-6416 [FREE Full text] [doi: 10.1073/pnas.1014748109] [Medline: 22460786]
31. Raykar V, Yu S, Zhao L, Valadez G, Florin C. Learning from crowds. The Journal of Machine Learning Research
2010;11:1297-1322 [FREE Full text]
32. Stein JA, Asman AJ, Landman BA. Characterizing and Optimizing Rater Performance for Internet-based Collaborative
Labeling. Proc Soc Photo Opt Instrum Eng 2011 Mar 3;7966 [FREE Full text] [doi: 10.1117/12.878412] [Medline: 21857775]
33. Khatib F, Cooper S, Tyka MD, Xu K, Makedon I, Popovic Z, et al. Algorithm discovery by protein folding game players.
Proc Natl Acad Sci U S A 2011 Nov 22;108(47):18949-18953 [FREE Full text] [doi: 10.1073/pnas.1115898108] [Medline:
22065763]
34. edX. URL: https://www.edx.org/ [accessed 2012-09-03] [WebCite Cache ID 6API7u8jc]
35. Taddei F, Sasco LR. Proposing measures to promote the education of creative and collaborative knowledge-builders. In:
Measuring Creativity: The Book. Brussels: European Commission; 2010:369-389.
36. Tuijn CJ, Hoefman BJ, van Beijma H, Oskam L, Chevrollier N. Data and image transfer using mobile phones to strengthen
microscopy-based diagnostic services in low and middle income country laboratories. PLoS One 2011;6(12):e28348 [FREE
Full text] [doi: 10.1371/journal.pone.0028348] [Medline: 22194829]
Abbreviations
DI: discrimination index
FP: false positive
TP: true positive
Edited by G Eysenbach; submitted 04.09.12; peer-reviewed by M Grobusch; comments to author 25.09.12; revised version received
08.10.12; accepted 24.10.12; published 29.11.12
Please cite as:
Luengo-Oroz MA, Arranz A, Frean J
Crowdsourcing Malaria Parasite Quantification: An Online Game for Analyzing Images of Infected Thick Blood Smears
J Med Internet Res 2012;14(6):e167
URL: http://www.jmir.org/2012/6/e167/
doi:10.2196/jmir.2338
PMID:23196001
©Miguel Angel Luengo-Oroz, Asier Arranz, John Frean. Originally published in the Journal of Medical Internet Research
(http://www.jmir.org), 29.11.2012. This is an open-access article distributed under the terms of the Creative Commons Attribution
License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any
medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete
bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information
must be included.
J Med Internet Res 2012 | vol. 14 | iss. 6 | e167 | p.13http://www.jmir.org/2012/6/e167/
(page number not for citation purposes)
Luengo-Oroz et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
