Conference PaperPDF Available

Amplifying prediction accuracy using Swarm A.I.

Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
1 | P a g e
Amplifying Prediction Accuracy using Swarm A.I.
Louis Rosenberg
Unanimous A.I.
San Francisco, CA, USA
Niccolo Pescetelli
University of Oxford
Clarendon, UK
Abstract In the natural world, many species amplify the
accuracy of their decision-making abilities by working together
real-time closed-loop systems that converge on optimal solutions
in synchrony. Known as Swarm Intelligence (SI), the process has
been deeply studied in schools of fish, flocks of birds, and swarms
of bees. The present study looks at the ability of human groups
to make decisions as an Artificial Swarm Intelligence (ASI) by
forming similar real-time closed-loop systems online. More
specifically, the present study tasked groups of sports fans with
predicting English Premier League matches over a period of five
weeks by working together in real-time swarm-based systems.
Results showed that individuals who averaged 55% accuracy
when working alone were able to amplify their accuracy to 72%
by forming real-time swarms. This corresponds to 131%
amplification in predictive accuracy across five consecutive
weeks (50 games).
Keywords Swarm Intelligence, Artificial Swarm Intelligence,
Collective Intelligence, Human Swarming, Artificial Intelligence.
Artificial Swarm Intelligence (ASI) strives to amplify the
collective wisdom of human groups by connecting participants
online into real-time closed-loop systems that are modeled
after biological swarms. Prior studies have shown that such
“human swarms” can produce significantly more accurate
predictions than traditional methods for tapping the collective
intelligence of groups, such as votes, polls, surveys, and
markets. For example, one recent study tested the ability of
human swarms to forecast the outcome of College Bowl
football games (in the U.S.) against the Las Vegas spread. A
swarm was comprised of 75 amateur football fans was tasked
with predicting each of 10 college bowl games. As
individuals, the participants averaged 50% correct (i.e. coin flip
accuracy). But, when working together as a real-time swarm,
those same participants achieved 70% accuracy against the
spread. Not only is this a significant accuracy increase, it also
enabled the 75 amateur football fans to out-predict the football
experts at ESPN [1].
While prior studies have documented the ability of
Artificial Swarm Intelligence to amplify the predictive ability
of online groups in singular events, no long-term study has
been performed to assess consistency of swarm-based
predictions over time. To address this, the present study tasked
human swarms with predicting all of the scheduled English
Premier League (EPL) matches over a period of five weeks in
2016. The objective was to assess whether or not a statistically
significant amplification of human intelligence could be
measured when comparing individual prediction accuracy to
swarm accuracy. In addition, swarm performance over the five
week period was compared to the predictions made by the
Sports Analytics Machine (SAM), a super-computer built by
the University of Salford to predict English Premier League
games using rigorous mathematical models [2]. Because SAM
results are published weekly by the BBC to reflect an expert”
assessment of weekly matches, this allowed for comparison of
professional level predictions with novice-based human
The decision-making processes in honeybee swarms have
been observed to be remarkably similar to the decision-making
processes in neurological brains [3,4]. Both employ large
populations of simple excitable units (i.e., bees and neurons)
that work in parallel to integrate noisy evidence, weigh
competing alternatives, and converge on decisions in
synchrony. In both, outcomes are arrived at through a real-time
competition among sub-populations of excitable units. When
one sub-population exceeds a threshold level of support, the
corresponding alternative is chosen. In honeybees, this enables
optimal decisions over 80% of the time [5,6,7]. It is this
amplification of intelligence that Artificial Swarm Intelligence
aims to enable among distributed networked humans.
Fig. 1. Usher-McClelland model of neurological decision-making
The similarity between neurological intelligence and swarm
intelligence becomes even more apparent when comparing
decision-making models that represent each. For example, the
Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
2 | P a g e
decision-making process in primate brains is often modeled as
mutually inhibitory leaky integrators that aggregate incoming
evidence from competing neural populations. A common
framework is the Usher-McClelland model [8] represented in
Figure 1 above. This can be directly compared to swarm-based
decision models, like the honey-bee model in Figure 2 below.
As shown, these swarm-based decisions follows a very similar
process, aggregating incoming evidence from sub-populations
of swarm members through mutual excitation and inhibition.
Fig. 2. Mutually inhibitory decision-making model in bee swarms
Unlike many other social species, humans have not evolved
the natural ability to form a closed-loop Swarm Intelligence.
Thats because we lack the subtle connections that other
organisms use to establish tight-knit feedback-loops among
members. Schooling fish detect vibrations in the water around
them. Flocking birds detect motions propagating through the
group. Swarming bees use complex body vibrations called a
Waggle Dance. Thus to enable a real-time Artificial Swarm
Intelligence among groups of networked humans, specialized
technology is required to close the loop among members.
To address this need, an online platform called UNU was
developed in 2015 to allow distributed groups of users to login
from anywhere around the world and participate in a closed
loop swarming process [9]. Modeled after the decision-making
of honeybee swarms, the UNU platform allows groups of
independent actors to work in parallel to (a) integrate noisy
evidence, (b) weigh competing alternatives, and (c) converge
on final decisions in synchrony, while also allowing all
participants to perceive and react to the changing system in
real-time, thereby closing a feedback loop around the full
population of participants.
As shown in Figure 3, participants in the UNU platform
answer questions by collectively moving a graphical puck to
select among a set of alternatives. Each participant provides
input by manipulating a graphical magnet with a mouse or
touchscreen. By positioning their magnet, users impart their
personal intent on the puck. The input from each user is not a
discrete vote, but a stream of vectors that varies freely over
time. Because the full population of users can adjust their
intent at every time-step (200 ms), the puck moves, not based
on the input of any individual, but based on the dynamics of
the full system. This enables real-time physical negotiation
among all members, empowering the group to collectively
explore the decision-space and converge on the most agreeable
solution in synchrony.
Fig. 3. A human swarm answering a question in real-time
It is important to note that participants do not simply vary
the direction of their input, but also modulate the magnitude of
their input by adjusting the distance between the magnet and
the puck. Because the puck is in continuous motion across the
decision-space, in order to apply force users need to
continually move their magnet so that it stays close to the
puck’s rim. This is significant, for it requires participants to be
engaged continuously during the decision process, evaluating
and re-evaluating their contribution. If they stop adjusting their
magnet to the changing position of puck, the distance grows
and their applied force wanes. Thus, like bees vibrating their
bodies to express sentiment in a biological swarm or neurons
firing activation signals to express sentiment in a neural-
network, the participants in an artificial swarm must
continuously express their changing preferences during the
decision process, or lose their influence over the collective
To assess the predictive ability of human swarms over an
extended period, a formal study was conducted over a five
week period using groups of randomly selected human subjects
from a pool of individuals who self-reported being enthusiasts
of EPL football. Each weekly group consisted of 25 to 31
participants who engaged the experiment via online access to
the UNU swarming platform. Each subject was paid $2.50 for
their participation in each weekly session, which required them
to make predictions for the outcome of all 10 English Premier
League matches being played that week, first as individuals on
a standard online survey, and then as part of a real-time
Artificial Swarm Intelligence comprised of the full weekly
group. In addition, the researchers compared results to the
predictions made by SAM, a sports super-computer at the
Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
3 | P a g e
University of Salford which uses ten years of data and
sophisticated algorithms to predict EPL games.
Across the full five week period, predictions were made for
a total of 50 games wherein the participants were required to
forecast one of three outcomes for each game: (i) Team A wins
the match, (ii) Team B wins the match, or (iii) the match ends
in a tie. It is worth noting that tie games occur at a rate of
approximately 25% in EPL matches, making it a significant
outcome possibility. It is also worth noting that 94% of the
swarm participants were American citizens for whom EPL is a
foreign sport covered mostly by foreign media. This context is
relevant when comparing performance of the human swarm to
the performance of the SAM super-computer, which is a UK-
based analytical system designed specifically to predict EPL
outcomes. In other words, it allows us to test if groups of
American fans, working together as artificial swarms, can
produce comparable results to a rigorous computational model
that is used by the BBC to forecast the UK’s national sport.
In Figure 2 below, a snapshot of a human swarm comprised
of 31 participants is shown in the process of predicting a match
between Arsenal and Watford. As shown in the figure, the
swarm is given five options to choose among, enabling the
swarm to identify which of the two teams will win and whether
the winning team will prevail by a single goal (“by 1”), by 2 or
more goals (“by 2+”), or if the swarm believes the match is too
close to call. In the example shown below, a large majority of
participants have already shifted their pull towards Arsenal,
and so the puck is currently heading in that direction.
Fig. 4. A human swarm predicting an EPL match in real-time
If the swarm converges on an answer that indicates one of
the two teams will win, that is selected as the predicted
outcome for the given match. If, on the other hand, the swarm
converges on “too close to call,” the swarm is given a second
question asking if the predicted outcome is most likely a tie. In
the example shown in Figure 4 above, the artificial swarm
demonstrated strong conviction that Arsenal would beat
Watford by a wide margin. In Figure 5, a series of snapshots
demonstrate how the swarm converged upon this final answer
over time. It should be noted that all predictions made by the
swarm were converged upon in under 60 seconds.
Fig. 5. A time-series of swarm converging on a final prediction
For each of the five weeks of the testing period, predictions
were made for the full slate of 10 matches that were played by
English Premier League teams. For each set of 10 matches, a
group of participants provided their individual predictions via a
Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
4 | P a g e
private online survey. The group also logged into the UNU
platform for real-time swarming and made predictions by
working together as a unified swarm. In addition, data was
collected from the BBC indicating the predictions made by the
SAM super-computer for the same games. After the games
were played, the results were scored by computing the number
of correct predictions and the percentage of correct predictions
for each test case. For individuals, the average values were
computed across the 25 to 31 participants in each group. These
results are shown in Table 1 below:
Table 1. Summary of prediction results over 5 weeks.
Assessing the raw results, we see that the swarm had the
best performance of the three experimental cases tested,
achieving 72% accuracy when predicting English Premier
League games. This was significantly more accurate than the
same individuals, when predicting independently, as they
averaged only 55% accuracy across each group. And finally,
the analytical super-computer, SAM, achieved a result in the
middle of these two cases, generating 64% accuracy.
Thus, at a first level of analysis we see that by working
together as a swarm, individuals who averaged 55% accuracy
when working alone were able to amplify their accuracy to
72% by forming real-time swarms and making the predictions
together. This corresponds to 131% amplification in predictive
accuracy across five consecutive weeks (50 games). This also
corresponds to a performance level that not only matched, but
slightly exceeded, an “expert source” of game predictions, the
SAM super-computer used by the BBC to publish expert picks.
Thus, by forming artificial swarms of approximately 30
individuals, groups of EPL fans (mostly American) were able
to make game predictions at an expert level.
To assess statistical significance, we compared the swarm
performance to the performance that would be expected by
chance from a matching population using a bootstrap approach
as follows: each week, we took a random sample of 10
individuals who participated in that week’s trial and took the
first individual's prediction for the first match, the second
individual's prediction for the second match and so on until we
had ten predictions from the ten randomly selected individuals.
We then averaged the accuracy of these predictions. We
repeated the procedure (i.e. random selection of ten individuals
and response assignment) 10000 times and computed the
average distribution of correct answers for that week.
Distributions are shown in Figure 6 below. The mean of the
distribution represents the average number of correct
predictions that should be expected by chance, by a matching
forecasters population. It can be seen that swarms are well
above the mean as compared to individual predictions. We then
computed the distance of the swarm performance for each
week from that week's mean in the form of a z-score distance
and computed the value of the cumulative density function of a
normal distribution with that mean and standard deviation. The
value indicates the probability of obtaining the score of the
swarm by chance.
Fig. 6. Individual vs Swarm predictions, assessed weekly.
To aggregate the results from the five weeks into one, we
compared the overall number of hits made by the swarm in the
5 weeks and the number of hits made by the average individual
(rounded to the closest integer). We then used a two-proportion
z-test, with the null hypothesis that the two hit rates are the
same. A z-statistic was obtained using the following formula:
z=(pIND pSWARM) / sqrt(p*(1-p)*(2/50))
where pIND is the hit rate of the average individual,
pSWARM is the hit rate of the swarm and p is the total sum of
hits made by both the average individual and the swarm and
divided by the total number of predictions (i.e. 100). The result
show that the average individual was significantly worse than
the unified swarm intelligence (z=-1.78, p=.03). The
aggregated results can be shown in a single profile, as depicted
in Figure 7 below. The red line indicates the superior
performance of the human swarm as compared to the
individual forecasters.
Fig. 7. Individual vs Swarm predictions aggregated for all five weeks.
Intelligent Systems Conference 2017
7-8 September 2017 | London, UK
5 | P a g e
Can swarms of novice participants such as casual sports
fans rival the predictive abilities of a respected expert source?
The results presented herein suggest this may be the case. As
demonstrated across five consecutive weeks of EPL match
predictions, swarms of approximately 30 average sports fans
were able to achieve competitive results to the SAM super-
computer that the BBC employs for providing expert level
predictions to the public. In fact, the 30 average sports fans,
when working together as an Artificial Swarm Intelligence,
out-predicted the SAM super-computer in four of the five
weeks. Even more significant, by thinking together as a
unified swarm intelligence, the groups of approximately 30
casual sports fans were able to significantly amplify their
collective performance across all five weeks of EPL match
predictions, boosting their overall prediction accuracy by 131%
as compared to the average individual participant.
Thanks to David Baltaxe and Chris Hornbostel of
Unanimous A.I. for their significant efforts in supporting the
data collection for this research study. Thanks also to
Unanimous A.I. for the use of the platform throughout
this five week experiment.
[1] Rosenberg, Louis. Artificial Swarm Intelligence vs Human Experts,
Neural Networks (IJCNN), 2016 International Joint Conference on.
IEEE. J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd
ed., vol. 2. Oxford: Clarendon, 1892, pp.6873.
[2] McHale, Ian. Boshnakov, Georgi, and Kharrati, Tarak. A Bivariate
Weibull Count Model for Forecasting Association Football Scores.
Working Paper 2016.
[3] Seeley T.D, Buhrman S.C 2001 Nest-site selection in honey bees: how
well do swarms implement the ‘best-of-N’ decision rule?. Behav. Ecol.
Sociobiol. 49, 416427
[4] Marshall, James. Bogacz, Rafal. Dornhaus, Anna. Planqué, Robert.
Kovacs, Tim. Franks, Nigel. On optimal decision-making in brains and
social insect colonies. Soc. Interface 2009.
[5] Seeley, Thomas D., et al. "Stop signals provide cross inhibition in
collective decision-making by honeybee swarms." Science 335.6064
(2012): 108-111.
[6] Seeley, Thomas D. Honeybee Democracy. Princeton Univ. Press, 2010.
[7] Seeley, Thomas D., Visscher, P. Kirk. Choosing a home: How the scouts
in a honey bee swarm perceive the completion of their group decision
making. Behavioural Ecology and Sociobiology 54 (5) 511-520.
[8] Usher, M. McClelland J.L 2001 The time course of perceptual choice:
the leaky, competing accumulator model. Psychol. Rev. 108, 550592
[9] Rosenberg, L.B., “Human Swarms, a real-time method for collective
intelligence.” Proceedings of the European Conference on Artificial Life
2015, pp. 658-659
... Examples of methods for tapping into and eliciting collective tacit knowledge are the aggregation of intuitions of multiple individuals, either by combining independent predictions through Delphi studies (Dalkey, Norman, and Helmer, 1963); crowd predictions (Hallin et al., 2016;Hallin et al., 2017Hallin et al., , 2019Tedlock and Gardner, 2015); swarm intelligence (Rosenberg et al., 2017); or prediction markets (e.g., Pennock et al., 2001;Servan-Schreiber and Wolfers et al., 2004). The term "prediction" is often used to refer to an informed guess or opinion by a group or crowd of people. ...
... The puck is generated by a central server and modeled as a real-world physical system with a defined mass, damping, and friction. Each participant in the swarm connects to the server and is allocated a controllable graphical magnet that allows the user to freely apply force vectors on the puck in real-time (Rosenberg and Pescetelli, 2017). Unanimous AI is an American technology company that provides ASI technology. ...
... At the organizational multi-team level, such as when policymakers carry out new policy reforms but tap into different groups' judgments using CI methods. An example is the adoption of swarm intelligence, such as when each participant connects to the server and is allocated a controllable graphical magnet that allows the user to freely apply force vectors on the puck in real-time (Rosenberg and Pescetelli 2017). It could also be citizens' deliberation on participatory platforms, supported by natural language processing (NLP) to sort and categorize the open-text contributions of interested citizens. ...
... Swarm is an online platform that allows users to interact concurrently to make predictions and answer questions. The design of the Swarm interface is modeled after biological swarms, and enables groups of online users to individually pose questions (selected by a moderator) and collectively answer these questions, make decisions, and resolve dilemmas by working together in real-time (Rosenberg and Pescetelli 2017). The platform is synchronous, meaning that users can explore decisionspaces together; furthermore, the software structures these online social groups of users through a process of "social swarming" in real-time, which is intended to promote group convergence on a preferred solution (WebFigure 1). ...
... Individual intent for selecting a particular answer was visible to others by showing small "graphical magnets" on the screen pulling the puck in specific directions (ie intended choice). In addition, participants were able to modify the direction of their intent continuously as they were observing the moving puck (see Rosenberg and Pescetelli [2017] for details). The group was given just 1 minute to reach a single answer; if consensus had not been achieved within that time, the question was asked again until a unanimous decision was reached (WebFigure 1). ...
... We defined results from the crowd as the aggregation of survey responses (ie mean for numeric estimation tasks and mode for multiple-choice tasks) for each question. For the swarm exercise, the aggregation of responses was defined as a consensus reached by the online swarm participants for each question through the use of dynamic graphics that facilitated the convergence of a swarm on a collective response (see Rosenberg and Pescetelli [2017] for details). With regard to guessing extremes, both the WOC and SI approaches performed well. ...
Incorporating relevant stakeholder input into conservation decision making is fundamentally challenging yet critical for understanding both the status of, and human pressures on, natural resources. Collective intelligence (CI ), defined as the ability of a group to accomplish difficult tasks more effectively than individuals, is a growing area of investigation, with implications for improving ecological decision making. However, many questions remain about the ways in which emerging internet technologies can be used to apply CI to natural resource management. We examined how synchronous social‐swarming technologies and asynchronous “wisdom of crowds” techniques can be used as potential conservation tools for estimating the status of natural resources exploited by humans. Using an example from a recreational fishery, we show that the CI of a group of anglers can be harnessed through cyber‐enabled technologies. We demonstrate how such approaches – as compared against empirical data – could provide surprisingly accurate estimates that align with formal scientific estimates. Finally, we offer a practical approach for using resource stakeholders to assist in managing ecosystems, especially in data‐poor situations.
... Recently, a new method has been developed that is not based on aggregating input from isolated individuals but involves synchronous groups of forecasters working together as real-time systems. Known as Artificial Swarm Intelligence (ASI) or Swarm AI, this method has been shown in numerous studies to significantly increase the accuracy of group forecasts [5][6][7][8][9][10][11][12][13]. ...
... These results add to previous research demonstrating that human groups can use Swarm AI to make better collective assessments across a wide range of domains, from subjective judgements and medical diagnoses to market forecasting [5][6][7][8][9][10][11][12][13][14]. While this study was limited in that it only involved forecasting volatile cult stocks with groups of MBA students, the results support prior research showing success amplifying the accuracy of group financial forecasts using Swarm AI [12]. ...
Conference Paper
Full-text available
Swarm Intelligence (SI) is a natural process that has been shown to amplify decision-making accuracy in many social species, from schools of fish to swarms of bees. Artificial Swarm Intelligence (ASI) is a technology that enables similar benefits in networked human groups. The present research tests whether ASI enables human groups to reach more accurate financial forecasts. Specifically, a group of MBA candidates at Cambridge University was tasked with forecasting the three-day price change of 12 highly volatile assets, a majority of which were cult (or meme) stocks. Over a period of 9 weeks, human forecasters who averaged +0.96% ROI as individuals amplified their ROI to +2.3% when predicting together in artificial swarms (p=0.128). Further, a $5,000 bankroll was managed by investing in the top three buy recommendations produced each week by ASI, which yielded a 2.0% ROI over the course of the 9-week study. This suggests that swarm-based forecasting has the potential to boost the performance of financial traders in real-world settings.
... In human groups, the technology of Artificial Swarm Intelligence (ASI) enables similar benefits by connecting networked groups as real-time closed-loop systems. Often referred to as "human swarms" or "hive minds", these systems have been shown to significantly increase accuracy in a variety of tasks, from predicting sports and equity markets to dispute resolution and medical diagnosis [1][2][3][4][5][6][7][8][9]. ...
... While ASI has been shown to amplify the accuracy of human groups in analytical tasks like forecasting, prioritizing, estimating, and diagnosing [1][2][3][4][5][6][7][8], formal studies investigating the potential of ASI to amplify the social intelligence of teams have not been conducted. This is important to scholarship as a group's mean social intelligence has been found to be a strong indicator of a team's overall performance [10,11]. ...
Full-text available
Artificial Swarm Intelligence (ASI) is a method for amplifying the collective intelligence of human groups by connecting networked participants into real-time systems modeled after natural swarms and moderated by AI algorithms. ASI has been shown to amplify performance in a wide range of tasks, from forecasting financial markets to prioritizing conflicting objectives. This study explores the ability of ASI systems to amplify the social intelligence of small teams. A set of 61 teams, each of 3 to 6 members, was administered a standard social sensitivity test —"Reading the Mind in the Eyes” or RME. Subjects took the test both as individuals and as ASI systems (i.e. “swarms”). The average individual scored 24 of 35 correct (32% error) on the RME test, while the average ASI swarm scored 30 of 35 correct (15% error). Statistical analysis found that the groups working as ASI swarms had significantly higher social sensitivity than individuals working alone or groups working together by plurality vote (p<0.001). This suggests that when groups reach decisions as real-time ASI swarms, they make better use of their social intelligence than when working alone or by traditional group vote.
... The assumptions of the model under discussion have been themselves derived from the data (see Section 5), but at the same time they make perfect sense in terms of modern economic theory. It is also worth noting that there is already an example of sporting events appearing [45] in research on a certain collective intelligence and the same technology then being successfully applied [46] in medicine. ...
Full-text available
We outline how to create a mechanism that provides an optimal way to elicit, from an arbitrary group of experts, the probability of the truth of an arbitrary logical proposition together with collective information that has an explicit form and interprets this probability. Namely, we provide strong arguments for the possibility of the development of a self-resolving prediction market with play money that incentivizes direct information exchange between experts. Such a system could, in particular, motivate simultaneously many experts to collectively solve scientific or medical problems in a very efficient manner. We also note that in our considerations, experts are not assumed to be Bayesian.
... Schumaker et al. [23] gathered tweet sentiments related to football and used it to predict outcomes and use as a wagering decision system. Rosenberg et al. [21] used a swarm intelligence model that amplified predictive accuracy when games were considered together as real-time swarms. Ryan et al. [22] analyzed the influence of maturity status and investigated the relationships between physical tests. ...
The English Premier League is the most competitive, televised, and widely watched football tournament. The research work discusses processing the historical match data and historical betting data for predicting the results of the future matches. The data processing consist of four stages, feature extraction, removing missing data, shuffling, and splitting dataset and experimented three models for match outcome and goal difference predictions. The outcome of the matches played between the competing teams were predicted using different models such as sequential minimal optimization (SMO), AdaBoost, and Bayesian networks to achieve the highest predictive accuracy. Data was collected in the form of historical match and betting statistical data for 12 seasons. The SMO attained 58.8% accuracy, the highest compared with other existing models. Correlation-based feature selection is used to indicate the most prominent features among match data and pointed out that Interwettens betting odds are better than others, having the highest correlation coefficient of 0.23 in comparison with the betting statistics of other companies. A new model is proposed using regression to calculate the goal difference to classify the match results in terms of home or away team win, or draw, with the results derived using the gradient boosting regressor model predicting the match outcome correctly in 97.4% of the matches. The accuracy of the proposed model, which achieved high entropy 0.96, indicates randomnesses of the statistical data. This new model uses supervised learning algorithm and achieved better accuracy in football match analytics, to assist the football players, coaches, for identifying their position and plan for further improvements.
... The SWARM Project and system have no relation to any other organizations or technologies using "SWARM" or similar terms. In particular, SWARM as used in this paper should not be confused with "SWARM AI" (Rosenberg & Pescetelli, 2017). For our project, the acronym SWARM initially stood for "Smartly Assembled Wiki-style Argument Marshalling," which described our initial bestguess as to what combination of crowdsourcing and SATs would have the best prospects for achieving CREATE's ambition. ...
How might analytic reasoning in intelligence reports be substantially improved? One conjecture is that this can be achieved through a combination of crowdsourcing and structured analytic techniques (SATs). To explore this conjecture, we developed a new crowdsourcing platform supporting groups in collaborative reasoning and intelligence report drafting using a novel SAT we call “Contending Analyses.” In this paper we present findings from a large study designed to assess whether groups of professional analysts working on the platform produce better-reasoned reports than those analysts produce when using methods and tools normally used in their organizations. Secondary questions were whether professional analysts working on the platform produce better reasoning than the general public working on the platform; and how usable the platform is. Our main finding is a large effect size (Cohen’s d = 1.37) in favor of working on platform. This provides early support for the general conjecture. We discuss limitations of our study, implications for intelligence organizations, and future directions for the work as a whole.
... The activity incorporated the use of Swarm AI technology [Rosenberg 2015] as a tool to answer a subset of questions from the first phase. Swarm is an online platform that allows users to interact concurrently to make predictions and answer questions [Rosenberg and Pescetelli 2017]. The platform is synchronous, meaning that users can explore decision-spaces together and the software structures these online social groups of users through a process of 'social swarming' in real-time intended to promote group convergence on a preferred solution. ...
Conference Paper
Full-text available
We present reports of two original research studies [Aminpour et al. 2020 and Gray et al. 2020] in which we used fisheries examples to empirically demonstrate how the Collective Intelligence (CI) of local fishing communities can be harnessed through pooling their Local Ecological Knowledge (LEK) to provide valuable information for sustainable resource management. In these examples, we used synchronous Social-Swarming technologies [Rosenberg 2015] and asynchronous Wisdom of Crowds [Surowiecki 2004] techniques for harnessing resource users’ CI to (a) estimate fish abundance, (b) predict human pressures on fish resources and (c) model complex human-fish interdependence.
Full-text available
The aggregation of individual personality tests to predict team performance is widely accepted in management theory but has significant limitations: the isolated nature of individual personality surveys fails to capture much of the team dynamics that drive real-world team performance. Artificial Swarm Intelligence (ASI), a technology that enables networked teams to think together in real-time and answer questions as a unified system, promises a solution to these limitations by enabling teams to take personality tests together and converge upon answers that best represent the group’s disposition. In the present study, the group personality of 94 small teams was assessed by having teams take a standard Big Five Inventory (BFI) test both as individuals, and as a real-time system enabled by an ASI technology known as Swarm AI. The predictive accuracy of each personality assessment method was assessed by correlating the BFI personality traits to a range of real-world performance metrics. The results showed that assessments of personality generated using Swarm AI were far more predictive of team performance than the traditional survey-based method, showing a significant improvement in correlation with at least 25% of performance metrics, and in no case showing a significant decrease in predictive performance. This suggests that Swarm AI technology may be used as a highly effective team personality assessment tool that more accurately predicts future team performance than traditional survey approaches.
Conference Paper
Full-text available
Groups often struggle to reach decisions, especially when populations are strongly divided by conflicting views. Traditional methods for collective decision-making involve polling individuals and aggregating results. In recent years, a new method called Artificial Swarm Intelligence (ASI or Swarm AI) has been developed that enables networked human groups to deliberate in real-time systems, moderated by artificial intelligence algorithms. While traditional voting methods aggregate input provided by isolated participants, Swarm-based methods enable participants to influence each other and converge on solutions together. In this study we compare the output of traditional methods such as Majority vote and Borda count to the Swarm method on a set of divisive policy issues. We find that the rankings generated using ASI and the Borda Count methods are often rated as significantly more satisfactory than those generated by the Majority vote system (p<0.05). This result held for both the population that generated the rankings (the "in-group") and the population that did not (the "out-group"): the in-group ranked the Swarm prioritizations as 9.6% more satisfactory than the Majority prioritizations, while the out-group ranked the Swarm prioritizations as 6.5% more satisfactory than the Majority prioritizations. This effect also held even when the out-group was subject to a demographic sampling bias of 10% (i.e. the out-group was composed of 10% more Labour voters than the in-group). The Swarm method was the only method to be perceived as more satisfactory to the "out-group" than the voting group.
Conference Paper
Full-text available
For well over a century, researchers in the field of Collective Intelligence have shown that groups can outperform individuals when making decisions, predictions, and forecasts. The most common methods for harnessing the intelligence of groups treats the population as a “crowd” of independent agents that provide input in isolation in the form of polls, surveys, and market transactions. While such crowd-based methods can be effective, they are markedly different from how natural systems harness group intelligence. In the natural world, groups commonly form real-time closed-loop systems (i.e. “swarms”) that converge on solutions in synchrony. The present study compares the predictive ability of crowds and swarms when tapping the intelligence of human groups. More specifically, the present study tasked a crowd of 469 football fans and a swarm of 29 football fans in a challenge to predict 20 Prop Bets during the 2016 Super Bowl. Results revealed that the crowd, although 16 times larger in size, was significantly less accurate (at 47% correct) than the swarm (at 68% correct). Further, the swarm outperformed 98% of the individuals in the full study. These results suggest that swarming, with closed-loop feedback, is potentially a more effective method for tapping the insights of groups than traditional polling.
Conference Paper
Full-text available
"Artificial Swarm Intelligence" (ASI) strives to amplify the combined intelligence of networked human groups by enabling populations of participants to form real-time closed-loop systems modeled after biological swarms. Prior studies [Rosenberg 2015] have shown that "human swarms" can converge on more accurate decisions and predictions than traditional methods for tapping the wisdom of groups such as votes and polls. To further explore the predictive ability of ASI systems, 75 randomly selected sports fans were assembled into real-time human swarms using the UNU software platform and were tasked with predicting College Bowl football games against the spread. Results show intelligence amplification.
Conference Paper
Full-text available
Much research has been done in the field of collective intelligence to aggregate input from human populations with the goal of amplifying the abilities of the groups. Nearly all prior research follows a similar model where input is collected from human participants in isolation and then aggregated statistically after the fact. The paper introduces a radically different approach in which the human participants is not aggregated statistically, but through a real-time dynamic control in which the participants act, react, and interact as a part of a system modeled after swarms in nature. Early testing of these "human swarms" suggest great potential for amplifying the intelligence of human groups, exceeding traditional aggregation methods. on the simulation of collaborative systems as it relates to the emergence of real-time collective intelligence. While theoretical studies are of great research value, there’s a growing need for real-world platforms that test the emergence of collective intelligence among human users. This short paper introduces such a platform. It enables networks of online collaborators to converge on questions, decisions, and dilemmas in real-time, functioning as a unified dynamic system. The dynamic system has been modeled after biological swarms, which is why refer to the process as “social swarming” or "human swarming". Early testing of human swarms suggests a great potential for harnessing collective intelligence.
Full-text available
The time course of perceptual choice is discussed in a model of gradual, leaky, stochastic, and competitive information accumulation in nonlinear decision units. Special cases of the model match a classical diffusion process, but leakage and competition work together to address several challenges to existing diffusion, random walk, and accumulator models. The model accounts for data from choice tasks using both time-controlled (e.g., response signal) and standard reaction time paradigms and its adequacy compares favorably with other approaches. A new paradigm that controls the time of arrival of information supporting different choice alternatives provides further support. The model captures choice behavior regardless of the number of alternatives, accounting for the log-linear relation between reaction time and number of alternatives (Hick's law) and explains a complex pattern of visual and contextual priming in visual word identification.
Full-text available
This study views a honey bee swarm as a supraorganismal entity which has been shaped by natural selection to be skilled at choosing a future home site. Prior studies of this decision-making process indicate that swarms attempt to use the best-of-N decision rule: sample some number (N) of alternatives and then select the best one. We tested how well swarms implement this decision rule by presenting them with an array of five nest boxes, only one of which was a high-quality (desirable) nest site; the other four were medium-quality (acceptable) sites. We found that swarms are reasonably good at carrying out the best-of-N decision rule: in four out of five trials, swarms selected the best site. In addition, we gained insights into how a swarm implements this decision rule. We found that when a scout bee returns to the swarm cluster and advertises a potential nest site with a waggle dance, she tunes the strength of her dance in relation to the quality of her site: the better the site, the stronger the dance. A dancing bee tunes her dance strength by adjusting the number of waggle-runs/dance, and she adjusts the number of waggle-runs/dance by changing both the duration and the rate of her waggle-run production. Moreover, we found that a dancing bee changes the rate of her waggle-run production by changing the mean duration of the return-phase portion of her dance circuits. Differences in return-phase duration underlie the impression that dances differ in liveliness. Although a honey bee swarm has bounded rationality (e.g., it lacks complete knowledge of the possible nesting sites), through its capacity for parallel processing it can choose a nest site without greatly reducing either the breadth or depth of its consideration of the alternative sites. Such thoroughness of information gathering and processing no doubt helps a swarm implement the best-of-N decision rule.
Full-text available
This study considers the mystery of how the scout bees in a honey bee swarm know when they have completed their group decision making regarding the swarm's new nest site. More specifically, we investigated how the scouts sense when it is appropriate for them to begin producing the worker piping signals that stimulate their swarm-mates to prepare for the flight to their new home. We tested two hypotheses: "consensus sensing," the scouts noting when all the bees performing waggle dances are advertising just one site; and "quorum sensing," the scouts noting when one site is being visited by a sufficiently large number of scouts. Our test involved monitoring four swarms as they discovered, recruited to, and chose between two nest boxes and their scouts started producing piping signals. We found that a consensus among the dancers was neither necessary nor sufficient for the start of worker piping, which indicates that the consensus sensing hypothesis is false. We also found that a buildup of 10–15 or more bees at one of the nest boxes was consistently associated with the start of worker piping, which indicates that the quorum sensing hypothesis may be true. In considering why the scout bees rely on reaching a quorum rather than a consensus as their cue of when to start preparing for liftoff, we suggest that quorum sensing may provide a better balance between accuracy and speed in decision making. In short, the bees appear to begin preparations for liftoff as soon as enough of the scout bees, but not all of them, have approved of one of the potential nest sites.
Full-text available
Stop, Don't Go There Decision-making in complex brains requires the reaching of a consensus based on information relayed by multiple neurons. A similar situation occurs in the decision-making process of swarming bees, where multiple individuals relay information about suitable hive sites, but a single site is chosen by the swarm. Seeley et al. (p. 108 , published online 8 December; see the Perspective by Niven ) show that consensus in this system is reached as information from scouting bees accumulates within the hive. Stop signals were given by scout bees from a particular site to those scout bees signaling from other sites. As the signals accumulate, scouting ceases and the bees prepare to swarm to the site that was best represented among the scouts. When this process was simulated, the results indicated that cross-inhibition among the bees functions similarly to that which occurs among neurons within complex brains. In both cases, such cross-inhibition prevents the overall system from coming to an impasse.
Full-text available
The problem of how to compromise between speed and accuracy in decision-making faces organisms at many levels of biological complexity. Striking parallels are evident between decision-making in primate brains and collective decision-making in social insect colonies: in both systems, separate populations accumulate evidence for alternative choices; when one population reaches a threshold, a decision is made for the corresponding alternative, and this threshold may be varied to compromise between the speed and the accuracy of decision-making. In primate decision-making, simple models of these processes have been shown, under certain parametrizations, to implement the statistically optimal procedure that minimizes decision time for any given error rate. In this paper, we adapt these same analysis techniques and apply them to new models of collective decision-making in social insect colonies. We show that social insect colonies may also be able to achieve statistically optimal collective decision-making in a very similar way to primate brains, via direct competition between evidence-accumulating populations. This optimality result makes testable predictions for how collective decision-making in social insects should be organized. Our approach also represents the first attempt to identify a common theoretical framework for the study of decision-making in diverse biological systems.
The paper presents a forecasting model for association football scores. The model uses a Weibull-inter-arrival times based count process and a copula to produce a bivariate distribution for the number of goals scored by the home and away teams in a match. We test it against a variety of alternatives, including the simpler Poisson distribution-based model and an independent version of our model. The out-of-sample performance of our methodology is illustrated first using calibration curves and then in a Kelly-type betting strategy that is applied to the pre-match win/draw/loss market and to the over-under 2.5 goals market. The new model provides an improved fit to data compared to previous models and results in positive returns to betting.
The time course of perceptual choice is discussed in a model of gradual, leaky, stochastic, and competitive information accumulation in nonlinear decision units. Special cases of the model match a classical diffusion process, but leakage and competition work together to address several challenges to existing diffusion, random walk, and accumulator models. The model accounts for data from choice tasks using both time-controlled (e.g., response signal) and standard reaction time paradigms and its adequacy compares favorably with other approaches. A new paradigm that controls the time of arrival of information supporting different choice alternatives provides further support. The model captures choice behavior regardless of the number of alternatives, accounting for the log-linear relation between reaction time and number of alternatives (Hick's law) and explains a complex pattern of visual and contextual priming in visual word identification.