LAPPEENRANTA-LAHTI UNIVERSITY OF TECHNOLOGY LUT
School of Engineering Science
Master’s Program in Software Engineering
CURRENT LITERATURE ON HACKATHONS:
OUTCOME SUSTAINABILITY AND REMOTE PARTICIPATION
Examiners: Associate Professor Ari Happonen
Professor Jari Porras
Lappeenranta-Lahti University of Technology
School of Engineering Science
Master’s Program in Software Engineering
CURRENT LITERATURE ON HACKATHONS:
OUTCOME SUSTAINABILITY AND REMOTE PARTICIPATION
Master’s Thesis 2020
46 pages, 4 figures, 12 tables
Examiners: Associate Professor Ari Happonen
Professor Jari Porras
Keywords: Hackathon, outcome, sustainability, remote, online, digitalization
This thesis work is based on Systematic Literature Review on hackathon like events. Thesis
research focuses on outcome sustainability and remote participation during these events. A
general overview of hackathon related research topics and coverage relevance have been
conducted with potential future research suggestions in mind. Selected material has been
classified during the in-depth reading process and focus-related observations recorded for
further analysis. The most notable observations are the following: 1) Publications on post-
event walkthrough are the most common ones, 2) There is a decent amount of research on
motivation during the event, although it mainly focuses on participant motivators, 3)
Hackathons are commonly viewed as part of the learning process that in many cases have
similarities with Project-Based Learning and/or emphasise the importance of teamwork, 4)
Event outcome sustainability is little researched and has contradictory findings as well as 5)
Remote participation in hackathon like event is almost not studied and is mostly limited to
subjective sidenotes in research focusing on other subjects.
Lappeenrannan-Lahden teknillinen yliopisto LUT
School of Engineering Science
CURRENT LITERATURE ON HACKATHONS:
OUTCOME SUSTAINABILITY AND REMOTE PARTICIPATION
46 sivua, 4 kuvaa, 12 taulukkoa
Työn tarkastajat: Associate Professor Ari Happonen
Professor Jari Porras
Hakusanat: Hackathon, tulosten jatkuvuus, etäosallistuminen, online,
Tämä työ pohjautuu kirjallisuuskatsaukseen (Systematic Literature Review) hackathon-
tapahtumista. Työn fokuksena on tapahtumatulosten kestävyys sekä etäosallistuminen
mahdollisuus kyseessä oleviin tapahtumiin. Myös hackathon-tapahtumien olemassa olevaa
tutkimuksen kattavuutta ja suosiota tarkastellaan potentiaalisen lisätutkimuksen
kartoittamisen näkökulmasta. Kerätty kirjallisuus on luokiteltu perinpohjaisen läpikäynnin
yhteydessä ja työn fokukseen liittyvät havainnot kirjattu ylös. Tärkeimpiä havaintoja ovat:
1) Menneiden tapahtumien läpikäyntiin liittyvää tutkimusta on reilusti eniten, 2)
Osallistumismotivaatiota on tutkittu runsaasti, vaikkakin se kohdistuu pääsääntöisesti
osallistujien motivaattoreihin, 3) Hackathoneja tutkitaan myös kohtuullisesti osana
oppimisprosessia ja niillä on havaittu olevan yhtenäisyyksiä projektioppimisen kanssa, 4)
Tapahtumatulosten kestävyyttä on tutkittu vähän ja olemassa olevat johtopäätökset ovat
jokseenkin ristiriitaisia, sekä 5) Etäosallistumiseen liittyvä tutkimus hackathonien
kontekstissa puuttuu lähes kokonaan ja nojautuu tällä hetkellä lähinnä subjektiivisiin
sivukommentteihin ja yksittäisiin julkaisuihin.
TABLE OF CONTENTS
1 INTRODUCTION ................................................................................................ 4
1.1 STAKEHOLDERS IN HACKATHONS ........................................................................... 5
1.2 TYPES OF HACKATHON LIKE EVENTS ....................................................................... 6
1.3 FOCUS, GOAL AND DELIMITATIONS ......................................................................... 6
1.4 STRUCTURE OF THE THESIS ..................................................................................... 8
2 METHODOLOGY ............................................................................................... 9
2.1 SYSTEMATIC LITERATURE REVIEW .......................................................................... 9
2.2 INTERVIEWING EXPERTS IN THE FIELD ................................................................... 10
3 RESEARCH PROCESS ................................................................................... 11
3.1 DEFINING SEARCH KEYWORDS FOR LITERATURE ................................................... 11
3.2 COLLECTING LITERATURE ..................................................................................... 12
3.3 LITERATURE DATA PROCESSING AND CLASSIFICATION.......................................... 13
3.4 LITERATURE OVERVIEW AND ANALYSIS ................................................................ 15
3.5 CASE MATERIAL AND INTERVIEWS ........................................................................ 17
4 HACKATHONS IN LITERATURE ................................................................... 18
4.1 HACKATHONS AS PART OF THE LEARNING PROCESS .............................................. 18
4.1.1 Project- and Challenge-Based Learning ..................................................... 18
4.1.2 Collaboration and teamwork ....................................................................... 19
4.2 MOTIVATION, ENGAGEMENT AND COMMITMENT IN HACKATHONS........................ 19
4.2.1 Motivators to participate and perform in hackathons ................................. 20
4.2.2 Benefits of mentoring for participants ......................................................... 21
4.3 SUSTAINABILITY OF HACKATHON OUTCOMES ....................................................... 21
4.4 REMOTE/ONLINE PARTICIPATION AT HACKATHONS ............................................... 23
4.5 LITERATURE ANALYSIS ......................................................................................... 23
5 INTERVIEWS WITH HACKATHON ORGANIZERS ....................................... 26
5.1 RESPONDENT SELECTION AND INTERVIEW QUESTIONS .......................................... 26
5.2 THE CONTENT OF THE INTERVIEW ......................................................................... 27
5.3 INTERVIEW ANALYSIS ........................................................................................... 31
6 DISCUSSION .................................................................................................... 33
7 CONCLUSION .................................................................................................. 36
REFERENCES ......................................................................................................... 37
LIST OF FIGURES
Figure 1: Process of data processing and classification ....................................................... 14
Figure 2: Publication yearly distribution after 1st round of iteration ................................... 15
Figure 3: Number of publications by year and source that were selected for SLR ............. 16
Figure 4: Topic classification distribution ........................................................................... 17
LIST OF TABLES
Table 1: Preliminary search term results from the Scopus database ................................... 11
Table 2: Source comparison for different search terms ....................................................... 12
Table 3: Number of publications by database with search keyword "hackathon" .............. 13
Table 4: Classes and their inclusion criteria's. Green indicates RQ related classes. ........... 14
Table 5: Intrinsic and extrinsic motivators to participate in hackathon like events............. 20
Table 6: Role and experience of interviewees ..................................................................... 26
Table 7: Interview questions with references to classification and RQs ............................. 27
Table 8: Answers to Q1 related to commonalities between hackathons and PBL .............. 27
Table 9: Answers to Q2 related to the role of teamwork in the learning process ................ 28
Table 10: Answers to Q3 related stakeholder motivation in hackathons ............................ 28
Table 11: Answers to Q4 related to hackathon outcome sustainability ............................... 29
Table 12: Answers to Q5 related remote participation in hackathons ................................. 30
LIST OF SYMBOLS AND ABBREVIATIONS
CBL Challenge-Based Learning
COVID-19 Corona Virus Pandemic 2019
IEEE the Institute of Electrical and Electronics Engineers
LC Learning Cycle
MVP Minimum Viable Product
PBL Project-Based Learning
RC Radical Collocation
RQ Research Question
SLR Systematic Literature Review
WoS Web of Science
Hackathons were originated in the late 1990s with a close tie to Silicon Valley, where this
format was used for prototyping and problem-solving activities . The term was coined by
OpenBSD that organized a seven-day event in 1999 around the open-source operating
system with the same name , . Hackathons are time-bounded events where participants
gather together, form teams and work together on the project of interest –. These events
are often associated with software development , ,  but in practice cover also other
areas such as civic engagement , , hardware , , healthcare ,  and society
development and sustainability aspects .
The length of these events varies usually between 24 and 36 hours  but could be extended
up to a week , . Team sizes might differ quite a bit but usually consist of three to five
people , . The term hackathon was first used in the year 1999  and derives from a
combination of words “hacking” and “marathon” . Since the term is relatively new there
is no single definition for it nor the event structure . Hackathon like events have different
names that go with suffixes such as “-fest”, “-jam” and “-camp” , . Generally
speaking, hackathons are usually software-focused  and competitive events  that
drives innovations ,  and social interactions , . They encourage learning in
innovative ways , especially when organised in an educational context  as well as
delivering quick and effective results with a hint of Agile methodology . In an
educational context, hackathons are usually highly motivating for both the educator and the
students , events include some form of competitive gamification like elements  such
as developing games to inspire students to learn coding skills for themselves . Often,
these events are collaborative by nature, between e.g., the educational unit which facilitates
the event and different partner organizations, who offer the intriguing challenges for the
students , .
Hackathons usually start with a case owner(s) presenting their subjects and/or concepts
followed by participants deciding on the topic to continue with . Trainer et al.  have
formalised the following development phases: 1) group forming, 2) role negotiations, 3)
process refining and deciding on working style and 4) the actual project performing. Flores
et al.  go a bit more in detail by distinguishing phase 4 into the following stages:
idealizing, designing, prototyping, testing and launching the solution for a given challenge.
This final solution usually culminates in some form of MVP (Minimum Viable Product) by
the end of the event . Events are sometimes organized competitively with monetary
prices ,  or even funding for further development of the project outcomes .
Multiple studies have identified that often projects are not being finalized ,  possibly
due to limited time-based stress affecting negatively to the idea and/or concept innovation
process . Studies have also identified ,  that there is a relationship between the
duration of the event and quality of proposed solutions, suggesting to extend the duration to
at least 24 hours. It is common for organizers to cater food and drinks during the event 
while the necessary hardware, like laptops, are usually encouraged to be provided by
participants themselves .
1.1 Stakeholders in hackathons
When it comes to hackathon like events, stakeholders could be divided into three main
groups: organizers, case owners and participants. For student participants, hackathons are a
good opportunity to work closely with professionals and tackle together relevant social
issues . There are a wide range of profit ,  and non-profit  parties who might
be interested in involvement in these events. Some of the main examples of such domains
are corporations , educational institutions , government agencies , open-source
 and other non-profit communities . Hackathons have become increasingly more
common during the 2000s which accordingly sparked more interest from corporate setting
and start-up companies , . This increase in interest has popularised sponsoring by
companies which in itself have transitioned hackathons from philanthropic baseline to more
competitive and innovation-driven style . Hackathons are also noted to have potential
on influencing corporate culture or even used as a tool for position company as “savvy digital
workplace” . As of writing, hackathons continue to be organized by different
stakeholders  with some companies event publicly stating plans on continuing such
efforts . F-Secure, one of the active companies utilizing hackathons, has stated that
although both organizers and participants have been satisfied with the events, the actual
outcomes have rarely been exploited commercially .
1.2 Types of hackathon like events
There are multiple ways to distinguish different type of hackathon like events, some of which
could be applied simultaneously. One of the most general separation criteria is if the event
is closed or open for public. Closed events are more common for inter-organizational target
audience  where the focus is more on innovating and learning aspects . Internal
hackathons are practised in companies , higher education institutions  and even civic
organizations , . Hackathons have especially gained traction in tech companies,
where these events have been integrated to support software development . Briscoe et al.
 have proposed a classification method based on describing the purpose of event with a
question, making them either tech- or focus-oriented. Tech-oriented ones are tackling
specific challenges with oftentimes predefined characteristic and requirements. On the other
hand, focus-oriented are more open-ended and focusing for example on the social or
business-related issues . Douthard et al.  proposes another kind of method that
classifies hackathons into 1) community nurturing, 2) issue-oriented and 3) innovation
Despite being originated from software development related needs , hackathons are after
all just a utility for driving different solutions and these events have been later integrated
into more interdisciplinary and civically focused settings . The literature identifies 
hackathons with different goals in mind, such as industrial , educational , culturally
 and civically  oriented ones. In some instances, hackathons are covering specific
themes  such as targeting social , ,  and environmental issues ,
enhancing learning , ,  or dealing with existing online ,  and offline
communities , .
1.3 Focus, goal and delimitations
Hackathon like events probably wouldn’t be so popular without any significant advantages
involved. Literature suggests that one of the main factors seems to be learning new skills
during the event , . These skills can be generally divided into soft skills like
teamwork, adaptability and critical thinking, as well as hard skills such as coding and
practical knowledge . Another significant factor seems to be a process of innovating
something new and interesting , , , , , . Short timeframe of the event
creates a sense of rush and forces participants to act fast without so-called “stovepiping”
. Creativity seems to be the driving factor for innovation and hackathons are no
exception from that . Also, hackathons seem to provide “new and exciting opportunities
for education and research” .
Naturally, there are also some disadvantages to hackathon like events in existence. Although
the short time frame is considered to be a driving force for creativity and innovation ,
some studies point out that it might also discourage the actual learning process , ,
 and affect quality of the end-results . The reason is that in a competitive and
corporate-sponsored event, solutions seem to be rewarded more than the actual learning
process. In other words, the focus is on solution and competition rather than learning and
collaboration . This is simply a result of time optimization, where participants tend to
utilize skills they already possess . Some go as far as to criticize hackathons for technical
solutionism where a deep understanding of the issue or subject might be deficient .
Sometimes hackathons are thought to appreciate a specific set of skills more than others,
affecting negatively learning process of some participants , . Also, the lack of formal
event structure and feedback from instructors were mentioned as reasons for hindrance in
In either way, hackathon events are used for educational and innovative purposes where
learning and other benefits act as motivators to participate. While the motivation,
commitment and engagement during the events are relatively studied aspects, the long-term
effect and sustainability are not. The question arises, what happens after the event? Are these
newly obtained skills, knowledge and experience being forgotten or do they hold up in the
future? Remote participation is another characteristic that hasn’t received a lot of attention
from researchers. Digitalization by itself is moving event organization from traditional and
physical to more digital format. In addition to that, events like COVID-19 (Coronavirus
Pandemic 2019) will probably only accelerate that process and highlight the importance of
measures such as remote participation.
The goal of this study is to have a look at what topics have been studied regarding hackathon
like events while specifically focusing on the following Research Questions (RQ):
RQ1) Status of post-hackathon outcomes sustainability
RQ2) Remote approach as an option for participation in hackathons
1.4 Structure of the thesis
Section 2 describes the methodology used for studying the research questions of this thesis.
The research process is explained in section 3 on a step-by-step basis including all necessary
data collection requirements and inclusions criteria’s as well as data processing and
classification results. The interview process is also briefly described in section 3.5. Section
4 includes literature observations derived from the research process finalised with literature
analysis in section 4.5. Interviews were conducted in section 5 for validating observations
from literature as well as narrowing the gaps that didn’t receive sufficient attention in
previous research. General discussion on the topic of this thesis is in section 6 followed by
conclusions in section 7.
For this study, primary method for gathering the information was a Systematic Literature
Review (SLR). Additionally, expert interviews were used for validating literature
observations as well as getting a better understanding of under-researched topics.
2.1 Systematic literature review
The study on literature in this work is based on SLR, a method suggested by Kitchenhand et
al. . It is a systematic practice of collecting secondary data, evaluating research studies
as well as concluding findings based on that . The main goal of SLR is to find academic
publications that are as closely related to specific research question as possible.
The main stages of SLR are :
1. Defining a research question
2. Defining exclusion and inclusion criteria’s
3. Quality assessment
4. Data collection
5. Data analysis
Defining the research question is an obvious thing to do, but not always easy and clear in
practice. It has to be properly balanced to be not too broad nor too narrow to correctly fit the
scope of the work. The first version of the keywords is usually derived straight from the
research questions. Later keywords are often modified or/and altered for example based on
the expert interviews or screening of other related publications.
Before collecting the data, proper boundaries have to be defined such as exclusion and
inclusion criteria’s as well as quality standards of publications. Some examples:
• Publications in a specific language
• Limited timeframe
• No too short publications
• Publications have to peer-reviewed
When these matters are taken care of the actual data collection can begin. There are several
different academic databases available, therefore the ones that match research topic in
question are recommended to be chosen. Some variability in databases might be also
beneficial in form of acquiring information from more diverse sources. This lowers the risk
of being stuck in a disciplinary bubble. The last step of SLR is contacting actual qualitative
and quantitative analysis based on gathered data.
2.2 Interviewing experts in the field
Interviews with experts in the field might offer some invaluable insight when it comes to the
research topic. While SLR is preferred and will remain as the primary method in the study,
some of the topics in this work simply haven’t been researched enough to gather adequate
information for making any valid conclusions. RQ2, in particular, seems to fit that criteria
and will therefore be approached mainly via interviews.
Interviews are also helpful in cases where topics are well researched. They can be used as a
complementary source for verifying and validating conclusions derived from SLR as well as
bringing light on topics that didn’t receive a lot of attention in the literature. While hackathon
like events are growing in popularity , , there seem to be some gaps in the research
literature. Therefore, interviews with organizers that have a lot of undocumented first-hand
experience in the field are a promising source of insightful information for this work.
For the interviews to be academically reliable and credible they are recommended to be
conducted systematically with a standardized basis. The simplest way to achieve this is to
stick to the unifying question palette that will be presented to all interviewees. This will
make the interview results more comparable. Standardised open-ended interview is the
formal definition for this method. 
3 RESEARCH PROCESS
The research process consists of two separate parts: SLR and interviewing experts in the
field. SLR is the primary one and starts with refining search keywords for discovering
literature (section 3.1). Then comes the process of collecting the literature (section 3.2)
followed by data processing and classification (3.3). SLR is finalised with Literature
overview and analysis. The interview research process is described in section 3.5.
3.1 Defining search keywords for literature
The research questions have been already identified in section 1.3. By looking at them the
main keyword is derived to be a hackathon. Based on RQ1 additional keywords are deducted
to be as follows: outcome, sustainability and support. RQ2 on the other hand is a bit more
unclear but eventually remote and crisis have been selected. For the initial search
experimentation, Scopus has been chosen, since its collection is the largest one. Search has
been conducted with just the main keyword alone, as well as in combination with the
additional keywords using AND-operator. Search results show in Table 1.
Table 1: Preliminary search term results from the Scopus database
Include hackathon related results
hackathon AND remote*
Search results are related to robotics
hackathon AND crisis
Hoped for COVID-19 related publications but
resulted in research with crisis-related topics
hackathon AND sustain*
hackathon AND outcome
Include hackathon related results
hackathon and support*
Include hackathon related results
As seen in Table 1 just using the main keyword produces a reasonable and sustainable
number of results. RQ1 related additional keywords produce also decent results where some
of the publications are more or less related to the specific research question. On the other
hand, RQ2 related additional keywords don’t produce satisfying results. The number of
results is really low, and they barely related to the topic of this work. Exact search result
numbers by different sources are available in Table 2.
Table 2: Source comparison for different search terms
Web of Science
Hackathon AND remote*
Hackathon AND crisis
Hackathon AND sustain*
Hackathon AND outcome
Hackathon and support*
Since the main keyword by itself leads to a reasonable number of results, the decision has
been made not to use the additional keywords. Additional keywords are necessary when the
number of publications is sky high, which is not the case in this situation. Additionally, all
the publications resulting from using the additional keywords are included in the main
keyword search results anyway.
3.2 Collecting literature
Databases used for gathering publications for this thesis are the following:
• Scopus by Elsevier
• Web of Science (WoS), Core Collection
• the Institute of Electrical and Electronics Engineers (IEEE)
Above mentioned databases have been suggested by the associate professor and work
supervisor as the most suitable ones when it comes to the topic of this thesis. WoS has the
broadest topic coverage and contains publications from the beginning of the 20th century.
Scopus on the other hand is the largest one, containing around 30 000 titles at the time of
writing . Lastly IEEE has been chosen because its publications are the most related to
our research question and computer science in general. All three database show similar
search results as shown in Table 1. They all differ in the number of publications but share a
similar ratio in regard to the actual search terms. Comparison is shown in Table 3 where only
the main keyword has been used.
Table 3: Number of publications by database with search keyword "hackathon"
Number of results
Web of Science
Search results have been exported from databases with the following information about
• Source link
3.3 Literature data processing and classification
Exported data have then been imported to Microsoft Excel for further processing resulting
in a total of 885 publications. First, all duplicates were removed resulting in a decrease of
publications from original dataset down to 625. Then the first round of iteration has been
conducted where filtering happened based on the publication title analysis. The main purpose
of that was to get rid of publications that don’t focus primarily on hackathon like events. In
many instances’ hackathon is mentioned more as a sidenote while the actual work is mostly
about something else. The number of publications decreased to 211 after the first round of
iteration. See the visualisation of data processing and classification in Figure 1.
Figure 1: Process of data processing and classification
The second round of iteration was based on publication abstracts analysis. This resulted in a
further decrease of dataset down to 108 publications. All remaining publications were
downloaded for proper in-depth reading. During the full reading of publications, topic
classification has been conducted for further analysis according to Table 4. There is no limit
for assigning classes to publication. Topics that are specifically related to the research
question of this thesis are shaded in green colour.
Table 4: Classes and their inclusion criteria's. Green indicates RQ related classes.
Inclusion and classification criteria
The publication is about event walkthrough with lessons learned.
Research focuses on motivation, engagement and commitment during the
The research focus is on the post-hackathon impact on outcomes. Things like
future interest in the subject, the continuation of an event project, sustainability
of learned skills etc.
Publication at least partly discusses remote participation at hackathons
The publication contains information about hackathons in general E.g., event
types and formats, history, stakeholders etc.
Hackathons are studied as part of the learning process
The publication contains a discussion on how to organize hackathon like events
Classification (full paper)
2nd iteration (abstract analysis)
1st iteration (title analysis)
After removing duplicates
Importing data 885
A B C D E F G
Classes C and D are based on research questions and the rest were derived from reading the
collected publications. The basis for identifying classes was recognition of some common
themes in literature. The actual classification has been conducted simultaneously while
reading through the material. While preparing for starting the actual writing process and
rereading material notes, some minor changes to classifications have been made.
3.4 Literature overview and analysis
The qualitative analysis demonstrated that hackathons as a topic in academia is relatively
new with first publications dating back to 2007 as shown in Figure 2. Nevertheless, this topic
seems to be in an uptrend and on average gaining popularity on a year-to-year basis.
Figure 2: Publication yearly distribution after 1st round of iteration
Figure 3 visualises the occurrence of publications selected for SLR after 2nd round of
iteration. While differing in numbers, both Figure 2 and Figure 3, resemble each other in
relative yearly distribution indicating that the screening process has been conducted properly
and with expected results.
010 20 30 40 50 60
Figure 3: Number of publications by year and source that were selected for SLR
When it comes to topic classification there are some interesting findings. See the
classification results visualised in Figure 4. There is a big difference in the occurrence of
classes previously defined in Table 4. Class A is the most common one having 71
occurrences. Classes G, B, E and F are moderate having between 15 and 34 occurrences.
The lowest in prevalence are classes C and D, both having less than ten occurrences.
Prevalence of class A speaks for the fact that most publications on hackathons are focusing
on event walkthrough, in many cases in pair with lessons learned (class G). Therefore, it
could be concluded that while hackathons are receiving a decent amount of research interest
in general, the focus is primarily on specific events and their post-analysis rather than more
general and epistemic approach. Classes B, E, G and G on the other hand have received a
decent amount of attention while at the same time not being overly studied. There seem to
be potential for further research on these topics especially with proper focus on understudied
The most understudied topics can be derived from classes C and D. Both classes have
received the least amount of attention and are therefore recommended for further research
based on this analysis. As already suggested by preliminary search results in Table 1 the
long-term effect from hackathons have not received a lot of attention (class C). This is quite
0 5 10 15 20 25 30
Web of science
unfortunate since one of the main goals of these events is to learn new things. The lack of
extensive research on systematic methods might also cause some uncertainty for event
organizers which in itself affects participants as well.
Figure 4: Topic classification distribution
Remote participation in hackathon like events has been almost non-existent, at least based
on the published publications. Remote participation (class D) has not received almost any
attention. Most publications only briefly touch this topic, without focusing too much on it.
However, COVID-19 and similar global events in the future might drive more interest in this
topic around remote participation.
3.5 Case material and interviews
Interviewing is a secondary research method in this work aimed at narrowing the gaps left
from the primary method – SLR. Therefore, this part of the work is not supposed to be overly
structured nor complicated. Interview questions are refined based on the findings in literature
analysis (section 4.5). Interviewee candidate selection and interviewing process is described
in detail in the 5th paragraph.
Sum of A
Sum of G
Sum of B
Sum of E
Sum of F
Sum of C
Sum of D
4 HACKATHONS IN LITERATURE
This section will describe literature findings on topics related to classes B, C, D and F, that
were previously described in Table 4. In other words, these topics are the following:
“Hackathons as part of the learning ”, “Motivation, engagement and commitment in
hackathons”, “Sustainability of hackathon outcomes” and “Remote/online participation”
4.1 Hackathons as part of the learning process
There seems to be a decent amount of interest when it comes to learning part of hackathon
like events. Parallels have been drawn between hackathons and both, Project-Based Learning
and Challenge-Based Learning. However, collaboration and teamwork during hackathons
have received the most attention as a contributing factor towards participant learning.
4.1.1 Project- and Challenge-Based Learning
In parallel with the traditional curriculum, Project-Based Learning (PBL) has been used as
an alternative for students to apply theories from lectures into practice with a focus on
collaboration and multi-disciplinary integration . The popularity of PBL within the
engineering education made it possible for students to apply technical knowledge to actual
engineering projects . These projects are often related to real-world problems or
challenges provided by industry representatives . There are similarities between the PBL
approach and the way hackathons are conducted . In both cases project is being tackled
in teams and similarly, mentors are often along with the process. Research indicates that
mentors are necessary for achieving the desired goals and outcomes of the project , .
Properly supporting teams working on complex projects is perceived as one of the major
challenges . Additionally, studies show that mentoring can contribute to an increase in
interest towards project topic in question , .
Another common approach to learning is to utilize the Challenge-Based Learning (CBL)
framework . In general, it is quite similar to PBL , but with a focus on solving a
challenge. It emerged as “an approach to foster learning while solving real-world problems”
 and was originally targeted to hight schools but later evolved to better fit higher
education . The short timeframe for CBL and PBL frameworks force teams to “narrow
down ideas and quickly find solutions to be developed”  leading to a development of
skills such as problem-solving, project management and task prioritizing .
4.1.2 Collaboration and teamwork
Collaboration has become a subject of interest in education  and crucial skill to master
at 21st century , . Teamwork seems to be playing one of the major roles in both PBL
as well as CBL frameworks . It is a complex process  that is believed to trigger or at
least enhance a successful team model . Well-functioning and effective teams could lead
to members trying the fullest of their abilities . According to the study  there is a
positive relationship between teamwork and effective learning as well as teamwork and
engagement behaviour. Other research ,  show that collaborative learning leads to a
deep understanding of the topic in question and individual change. It is common at hackathon
like events that members of just formed teams are not familiar with each other and for these
situations, icebreaking techniques are recommended to bring individuals closer together
, . Team building practices are common for organizational development  and is
considered essential for establishing successful teams , . Leadership is an essential
foundation for effective teamwork –, both vertical and shared . There is clear
evidence of difficulties for teams to work efficiently and distribute tasks in an optimal way
, which hints at the importance of proper leadership. Besides, suitable membership,
commitment, argumentation and working methods are considered conditional for forming
interdependency within team members, which boosts learning , , . Setting a clear
goal and objectives seem to increase performance  and make the context more
meaningful to a learner .
4.2 Motivation, engagement and commitment in hackathons
Factors affecting participants motivation, engagement and commitment towards hackathon
like event have received relatively a lot of attention. A common approach to tackle this topic
is via identifying the participation motivators as well as the mentor’s role in creating and
supporting engagement during the event.
4.2.1 Motivators to participate and perform in hackathons
There is a huge amount of interest in making the education process more interesting and
engaging for students . While research on hackathon like events is relatively scarce,
multiple studies have been conducted to better understand the willingness of participants to
engage in such events , , , . The challenging part of these studies is that
motivating factors are presented indefinitely and non-informatively . Motivating factors
to participate in hackathon like events could be roughly divided into intrinsic and extrinsic
one , where some of the motivators can apply to both categories. See Table 5. There are
some geographical differences in the prevalence of different motivators , . For
example “hackathon participants in Europe and Asia agreed (88%) that they registered in
hackathons because they are fun events, while those from the United States disagree in a
proportion of 33%” . Hackathons participants have shown an increase in motivation
regarding programming and general computing as well as practical improvements in
practising above mentioned tasks .
Table 5: Intrinsic and extrinsic motivators to participate in hackathon like events
Fun and enjoyment , , –,
Job performance and career concerns ,
, , , , 
Experimenting with new technologies ,
Social influence , , , ,
Intellectual challenge , , , 
Financial gain , , 
Personal development , 
Status and reputation 
Interest in subject , 
Fair judgement system 
Innovating , 
Contributing to a social problem , 
Learning and developing skills , , , , , –
Networking , , , , 
Applying personal skills , 
Professional and personal identity , , 
Collaboration , 
4.2.2 Benefits of mentoring for participants
Mentoring, or in other words coaching, is considered to be a crucial factor for participants
to create their solutions . Mentors role is to support participants with the creation of
solution  and in general to guide into the right direction. There is an indication of bonding
between mentor and participants, which might positively affect project progression .
Well established communication leads to frequent and effective feedback which on itself
result in getting more work done , . There is evidence  that inter-community
feedback boosts commitment and self-esteem on an individual level. According to research
 longer and more intensive mentoring leads to the creation of stronger ties between
participant and mentor. Whereas people who develop these strong connections in groups,
seem to work harder, get more done, and tend to stick with the group for longer periods
. Other research supports that idea by pointing out that members who work on common
tasks tend to be more united and committed , . While working on these
interdependent tasks during the event, the participant may feel more important and therefore
commit more actively to the goal , whereas independent tasks are speculated to lower
the levels of commitment .
Mentoring is a complex process that can be both, successful and non-successful. Studies
show that in the context of hackathon like events, guiding and supporting participants in
seeking their solutions leads to more positive and engaging outcomes , , . On
the other hand, fixating to predefined goals without adjusting participants abilities might
lead to misunderstanding between mentor and participants and worse atmosphere . The
way hackathons are constructed might leads simultaneously to cooperation, competition or
both . These competitive setups with prizes, peer pressure and tight schedules encourage
and motivates participants to go with easy and effective solutions rather than focus on actual
individual learning .
4.3 Sustainability of hackathon outcomes
Hackathon outcomes are considered to be for example prototypes, networking, learning and
other direct results . While hackathons themselves are relatively studied, the long-term
effect of these activities very little, if at all . The little research that exists has a
contradictory interpretation of the matter. Some even suggest that hackathon outcomes are
not sustained at all , , , hinting at the uselessness of doing so. According to one
survey,  80% of hackathon participants expressed plans on continuing further
development while only one third actually achieving in this. Partly explained by the lack of
support by organizers , it still looks like more research is needed to gain a better
understanding. At the same time, other studies suggest that there is reason to believe that
hackathon like events may be “an effective way to put scientific software projects on
sustainable trajectories” , .
Continuation of a project beyond hackathon like event mostly depends on a market need as
well as the project’s fit to the already existing product, if that is the case . Proper
community is mentioned  to be one of the crucial ingredients for the success of
scientific software sustainability. The need for proper follow-up process is recognized with
a potential benefit of building new collaborations and in learning more about project
development processes . To support these follow-up processes, such as developing,
marketing and launching, it has been suggested to find the right people interested to sponsor
activities of such kind , . The importance of corporate representative in
hackathons is highlighted in several studies suggesting that they should be more active at
helping winners at finalizing their solutions and launching them to the market after the actual
event , . As an example, to sustain the continuation and development of artefacts
created in hackathon like events the following have been offered by several organizers:
Coaching and mentoring, showcasing the artefact during an event , prizes , ,
releasing a production version of artefact , recruiting new team members  and
writing grants .
Learning Cycles (LC)  have been applied to post-hackathon activity, where the aim is
to “connect hackathon teams with key stakeholders, reflect on prototypes and consider
business models” . LC is described as a “group of people who come together to engage
in dialogue about a common interest” where the process is marked by “equality and
empowerment of all participants” . While hackathons role as part of the learning
process is recognized , it seems like a complementary step, such as LC, might be useful
for better outcome sustainability .
4.4 Remote/online participation at hackathons
Primarily hackathon like events are organized in physical venues , but there are also
reports of virtual participation . The topic of remote participation in these events hasn’t
received a lot of attention as previously already hinted in section 3.4. The little research that
exists seems to showcase an attitude that is ranging from neutral to negative. According to
quantitative analysis of one study , the offline setting is slightly preferred participation
method over online platforms explained by interviewees “because it is not a real-life setting
and it could foster extreme or non-constructive opinions”. In another study , findings
indicate “difficulties related to remote mentoring which did not only appear to hinder
students to progress with their project but might also have affected their future interest in the
community”. This is a strong statement based on single-event experience, but none the less,
a real-world observation, which “resulted on average in poor learning experience (…) for
students” and “low-quality connection between the parties” . In one case , a hybrid
approach was used, where some of the participants were physically present while others
connected via online meeting tools. The online meetings are considered by different
practitioners to be inferior to physical ones .
Radical collocation (RC) means a situation where team members are located together in a
room for a period of the project . This methodology was developed in response to
communication-related challenges in distributes software development and it is noted to
increase coordination and communication among software team members, resulting in
“increase in productivity and outcome quality” .
4.5 Literature analysis
Quantitative analysis revealed several prominent findings concerning hackathon like events.
Both, Figure 2 and Figure 3 indicate that the general trend in publishing material related to
these events is still uprising as of writing. A local peak is identifiable in the year 2019, with
lower results in 2020. While impossible to know for certain, the reasons for that is likely
relative longevity of the research process as well as lagging in the actual publication. E.g.,
research is conducted in the second part of 2018, but the publication date is likely to be in
the following year 2019.
The publication classification conducted according to section 3.3 showcases some
interesting insight on the status of research in the area of hackathon like events. There is
quite a noticeable difference in research interest as seen in Figure 4. Class A, which is about
walkthrough of specific hackathon event, have received over twice as much attention that
any other class in Table 4. Classes B, E, F and G on the other hand have received a moderate
amount of attention while classes C and D extremely little. Reading through collected
publications from SLR, many of the above-mentioned points were supported. As already
suggested by Figure 4 publications are predominantly about post-analysis of recently held
events (class A), many of them in pair with lessons learner and suggestions for future similar
events (class G).
The topic of hackathon like events as part of the learning process (class F) have received a
decent amount of interest. Several pieces of research have brought up similarities between
hackathons and Project- and Challenge-Based Learning methodologies. Another common
dominator appears to be the importance of teamwork in learning with hackathons. Although
relatively studied topic, there is room to go for future research with proper targeting and
Although, specifically motivation focused publications (class B) received a decent amount
of attention, settling in the middle of Figure 4, multiple other publications touched that topic
as a sidenote. But despite the general interest, topics seemed to focus almost solely on
participants motivation to engage in hackathon like events. The motivators behind corporate
representation, organizers, problem owners and mentors received almost no attention, at
least compared to participants. In other words, there seem to be some potential to investigate
other stakeholders’ motivation without clinging too much to participants viewpoint.
The literature about post-hackathon outcome sustainability (class C) is scarce even for
hackathon standards, ranking second lowest in classification as seen in Figure 4. This topic
is directly related to the RQ1. The identified literature that exists is heterogeneous and one
could say even contradictory, ranging from extremely pessimistic attitude to belief in the
good potential of succeeding in outcome sustainability. There is real-life evidence of
attempts on sustaining hackathon outcomes via different methods, but very little has been
studied on the effectiveness of above-mentioned. This leads to a conclusion that the topic of
outcome sustainability has somewhat gained interest but haven’t been studied extensively as
of writing this work. Therefore, there seems to be a lot of potential for future research.
Remote participation, the topic of RQ2, is almost entirely avoided in hackathon related
research. There are almost no publications specifically focusing on this topic, leaving a lot
of room for uncertainty. The question arises if there is even any potential in this topic after
all, based on the low general interest from academia. The little research that exists has for
the most part negative attitude towards remote participation with some neutral ones amongst.
The situation looks bleak, but on the other side, it is immature to make any definitive
conclusions based on such little evidence and almost non-existent literature. One could even
speculatively assume, that this topic might gain some traction in the future as technology
progresses and people’s attitudes change. Also, maybe remote participation could work out
as a secondary or additional research topic by utilizing A/B testing methodology. E.g., A
hackathon research where subjects are divided into two groups: A) Offline and B) online
participant whose performance will be monitored and documented.
5 INTERVIEWS WITH HACKATHON ORGANIZERS
This section aims to compare findings from the literature review with the first-hand
experience by hackathon organisers. Additionally, interviews might be a valuable source of
information on topics that weren’t satisfactorily discussed in the literature as of writing this
thesis. Respondent selection and interview questions are discussed in section 5.1, interview
content is available in section 5.2 and interview analysis is conducted in section 5.3.
5.1 Respondent selection and interview questions
Interview candidates were selected with the assistance of thesis instructors. The criteria for
selection was at least a moderate amount of experience in organising hackathons, preferably
with a remote approach as well. Eventually, four candidates were interviewed anonymously.
See Table 6 for more information about respondents (R).
Table 6: Role and experience of interviewees
Role and experience in hackathons
Associate Professor at the faculty of Software Engineering.
Hackathon related academic research conducted. About 15 years of experience in
organizing hackathons. 2018 onwards experience in remote hackathons as well.
Professor at the faculty of Software Engineering.
Hackathon related academic research conducted. About 20 years of experience in
organizing hackathons, including remote ones past few years.
Co-founder and managing director at the company that organises hackathon event.
Several years of experience in organising both normal and remote hackathons,
mainly focusing on the corporate setting.
Associate Professor in Information Systems. Hackathon related academic research
conducted. Almost a decade worth of experience in organizing both normal and
The interview questions were generated based on the conducted literature analysis in section
4.5. These questions are either aimed at validating previously deducted observations or
alternatively exploratorily searching for entirely new answers to RQs. Therefore, final
questions are based on classes defined in Table 4 as well as RQs from section 1.3. See the
questions in Table 7. Referring to question with abbreviation Q as of now.
Table 7: Interview questions with references to classification and RQs
Have you noticed elements of PBL in hackathons?
What is the role of teamwork in learning process in the context of
Have stakeholder motivators been clarified (before the event)? Which
stakeholders? How? How much?
What factors can support and help to sustain event outcomes? How to
do that in practice?
General question on experience in remote hackathons. Differenced
between remote and normal approach? Advantages and disadvantages
in remote participation?
5.2 The content of the interview
This section will focus on the content of the interview. Interview questions are as in Table 7
with respondent references according to Table 6.
Table 8: Answers to Q1 related to commonalities between hackathons and PBL
Q1: Have you noticed elements of PBL in hackathons?
Hackathons can surely be organised utilising PBL, but it is not necessary and
depends on the approach. Hackathons aren’t always based on a problem or
challenge to solve but can be instead focused on having fun (e.g., game jams).
Except for the fact that PBL is supposed to be in pair with formal theory, it has
otherwise similarities with the way hackathons are conducted. It has to be noted
though, that limited timeframe of these events makes it almost impossible to satisfy
formal PBL formulation. But in general, hackathons are specifically based on a
problem or challenge to solve.
Not certain about the formal definition of PBL, but I can say that we have
implemented problem-solving in our hackathons. Not always though.
Hackathons and PBL are kind of the same thing, but the former is not usually
integrated with educational material. Also, there is rarely any control on what topic
participants want to work on as opposed to the educational environment.
Table 9: Answers to Q2 related to the role of teamwork in the learning process
Q2: What is the role of teamwork in learning process in the context of
We are encouraging teamwork in hackathons because participants can achieve
more that way. Teamwork increases social networks and lowers the barrier to
engage with other participants. Usually, teams have a more diverse set of skills
leading to a better chance of success. Solo approach, on the other hand, leads more
likely to dropping out.
Hackathons are based on both, inter- and intra-team collaboration. Teamwork is
important for learning, but not as important in a corporate setting. Learning and
collaboration seem to be correlated as opposed to the corporate and more
Our hackathons are always based on teamwork. It forces people to connect with
each other. Competitive environment seems to increase inter-team disagreements.
Typically learning is a social context and hackathons are fundamentally based on
Table 10: Answers to Q3 related stakeholder motivation in hackathons
Q3: Have stakeholder motivators been clarified (before the event)? Which
stakeholders? How? How much?
Motivation is mostly discussed in the context of participants since the whole
existence of event depends on the fact that people are motivated to participate. It is
a good practice to get feedback after the event (e.g., questionnaire). If the feedback
is positive organization is probably interested in continuations as well. Hard to
comment on corporate cooperation because motivation varies largely.
In learning-oriented hackathons, you have to think on what there is for participants
to be motivated. Technically oriented hackathons (e.g., code-camps) are pretty
straight forward. Learning new skills is the main motivator there. Customer side is
usually pretty clear as well (e.g., raising awareness on a specific topic). Corporate
hackathons are usually motivated by finding new recruits or contacts.
Focus is mostly on participants because problem owner motivation is usually pretty
clear. Nailing motivation is especially important at short events. Problem owner
(corporate context) motivation examples: increasing engagement, socialising and
Underlying motivators are generally the same for different stakeholders.
Unfortunately, companies are oftentimes unclear on motivation. E.g., companies
are rarely honest on their motivations (hiring, generating money, free business
ideas etc.) Organisers motivation is quite clear usually, but the mentors not so
Table 11: Answers to Q4 related to hackathon outcome sustainability
Q4: What factors can support and help to sustain event outcomes? How to do
that in practice?
It is important to think about outcome sustainability in advance. The event must be
structured in a way that supports the continuation of outcomes. E.g., Financial
support for winners, company assisting with continuation and hiring as an assistant
in education. There is also a potential for getting funding from several parties like
scholarships, grants, accelerator programs etc.
The topic is very little studied. Contacting customers afterwards and asking how
the outcomes have been supported. It might be helpful to ask for feedback from
participants as well.
Outcome sustainability is a bit problematic. If the problem statement is too broad, it
is often challenging to fit the outcome into the market. Therefore, the missions
statement has to be thought through beforehand and adjusted to goals. The best way
to support outcome sustainability is to mentor sessions after events. Sometimes
hackathon winners are promised assistance by clients with a continuation of the
Mostly event outcomes depend on participants themselves. Are they interested to
continue or not? Winning might work as a motivator to continue but doesn’t
necessarily lead to success and continuation of the project. The decision has to be
made on what the preferred outcome is, and the event should be planned
accordingly. E.g., If the desired outcome is to create a complicated app, focus on
proper teambuilding so participants could realistically achieve the goal.
Table 12: Answers to Q5 related remote participation in hackathons
Q5: What is your experience on remotely held hackathons? What is the
difference compared to normally organised hackathons? What are the
advantages and disadvantages?
Distribution of material in remotely held hackathons happens over video.
Companies have been interested because it is more practical for them. No need to
travel any more. Hybrid methods have been also used where both local and remote
participation is possible. The interest towards topic is growing, especially with the
coming of COVID-19. The remote approach is more challenging. The role of a
mentor/facilitator is increased. Communication in general is also challenging. E.g.,
unexpected dropping out of the event with no explanation. It is also possible that
inter-team collaboration is less common.
Little experience. The situation might change in near future because of the COVID-
19. Remote participation may decrease collaboration, which is problematic since it
is so essential in the context of hackathons.
The quality of the work is the same. Difference is mainly in the atmosphere – the
feeling is not there. It is less fun remotely but possibly more effective (no chit chat
etc.). The remote approach is better for one-to-one meetings if necessary. There are
some advantages. E.g., Having speakers and participants all over the world.
It is more difficult to control the event remotely. People might just disappear
suddenly. Team forming is also harder. The difference in time zones might be a
challenge as well. Mentoring takes more resources in remote events. The event
relies more on mentors/facilitators.
5.3 Interview analysis
The topic of hackathons as part of the learning process consist of two questions. The first
one (Table 8) asks if there are similarities between PBL and hackathon like events. Answers
to this one are pretty homogenous with a consensus on the fact that hackathons can and often
are organised around problem-solving. R2 and R4 recognize that hackathons are only partly
filling the definition of PBL since these events are rarely integrated with purely educational
material. The second question (Table 9) is about the role of teamwork in the learning process
in hackathon like events. All respondents seem to agree on the importance of teamwork with
benefits ranging from outcome efficiency (R1) to better learning (R2). There is also an
interesting observation by R2 and R3 acknowledging the negative effect of corporate and
competitive environment on collaboration.
Hackathon literature seemed to mainly focus on participants motivation to engage with the
event and other stakeholders received less attention. The question arises where this
difference is coming from. R1 justify it with an observation that the mere existence of these
events depends on interest by participants. R2 and R3 also mention that the case owner’s
motivation is usually pretty clear to begin with, hence the focus specifically on the
participant side. R4 on the other hand brings up a problem of unclear motivators of case
owners. Companies rarely directly explain their motivation to engage in hackathons. E.g.,
The true reasons might be hiring new people, generating money and finding free business
Outcome sustainability is very little studied (R2) so getting first-hand expertise from
hackathon event organisers might be very valuable. When it comes to answers on Q4 there
seems to be one common feature amongst respondents: Post-outcome sustainability depends
on properly adjusted mission statement (R3) and structuring event in a way that supports
these outcomes (R1 and R4). R1 emphasise planning the event having outcome sustainability
in mind, while R2 expressed the risk of too broad problem statement leading to a challenge
on fitting the outcome to the practice. Some of the proposed solutions for supporting
outcomes are the following: Money prizes (R1), assistance in outcome continuation (R1, R3)
and post-event mentoring sessions (R3). R2 suggested that organisers could collect feedback
from participants as well as problem/challenge owners, effectively leading to potentially
better outcomes in future events.
Respondents had mixed feeling towards remote participation in hackathons (Table 12)
having both positive and negative things to say. The remote approach seems to be more
practical for company representatives as the need to travel diminishes with online attendance
(R1). It also makes international events possible (R3) albeit difference in time zones might
be a challenge (R4). However, there are also some downsides such as a huge negative change
in the atmosphere (R3), decrease in collaboration (R1, R2) and increased risk of sudden
dropouts during the event (R1, R4). Remotely held events also seem to demand more
mentoring resources (R1, R4). Although little studies (R2) it has been speculated that
COVID-19 might cause an increase in interest towards a remote approach in hackathons
(R1, R2). It appears to be also possible to organise hybrid hackathons with optional and fully
remote participation option (R1).
Systematic Literature Review (SLR) was used as a primary method for collecting material
and conducting initial quantitative analysis. During the process total of 885 publications
were collected from three different academic publication databases: Scopus, WoS and IEEE.
After removing the duplicates and two rounds of preliminary screening based on both, title
and abstract analysis, 108 publications were left for topic classification that was conducted
during in-depth reading according to criteria shown in Table 4. Classification results, visible
in Figure 4, indicated a tremendous difference in topic relevance ranging from only three to
as high as 71 class-specific assignments.
Several observations were deducted from the in-depth walkthrough of selected literature.
Publication centring around hackathon like event walkthrough (class A) were the most
common ones. Following topics got a decent amount of traction from academia as well:
motivation, engagement and commitment during the hackathon like events (class B), general
hackathon related walkthrough (class E), hackathons as part of the learning process (class F)
and exploration on how to organise hackathons (class G). Unfortunately, post-hackathon
outcome sustainability (class C), as well as remote participation at hackathon like events
(class D), received very little attention.
Despite being studied, classes B, E, F and G have the potential to be researched more with
proper topic targeting and segmenting. E.g., Class B related studies concentrate on
participant motivators to engage in hackathons leaving other stakeholders, like organizers
and case owners, in a vacuum. Finally, classes C and D carry huge potential regarding future
research. Both topics are extremely understudied with contradictory interpretations (class C)
and seemingly anecdotal evidence (class D). E.g., Research has a varying attitude towards
post-hackathon outcome sustainability ranging from extremely pessimistic to a somewhat
promising one. Also, feedback on remote participation in such events is mostly limited to
negative and subjective comments.
Literature findings from section 4 explained the current status of research on the topic of
hackathon like events as of writing this thesis. Some of the topics received more attention
than others partially leaving uncertainty in the air. It was decided to additionally conduct
anonymised interviews with hackathon event organizers in an attempt to narrow that gap in
uncertainty, compare literature with real-life experience and possibly discover entirely new
findings. The four respondents that were interviewed are introduced in Table 6 and interview
questions listed in Table 7.
Expert interviews brought some insight to the above-mentioned observations from literature
analysis. E.g., The reason for mainly focusing on participants motivation to participate in
hackathons simply appears to be a considerably better understanding of other stakeholders’
motives. Compared to literature, interviews provided especially significant breakthroughs to
the research questions of this thesis. There was a strong view amongst respondents that to
sustain hackathon outcomes (RQ1), the mission statement should be adjusted properly, and
event structured with the outcomes in mind in advance. Some practical suggestions were
also provided, matching ones in the literature. Respondents shared similar attitudes towards
remote participation in hackathons (RQ2) compared to literature findings, although they
were a bit more optimistic about it. In spite of some clear advantages to the remote approach,
like practicality of online attendance and opportunity for more international events, there are
also some disadvantages, like requiring more mentoring recourses and lack of notorious
hackathon affiliated atmosphere. As of writing this thesis, several respondents noted a
potential increase in interest toward remote participation because of currently ongoing
COVID-19 related events.
One thing that personally stood out was a notion of seemingly dualistic attitude towards
motivations and characteristic around organizing hackathons. It seems like the focus on these
events is either on A) collaboration, teamwork and learning, or B) competitive and
commercially centric innovation. These two types don’t appear to be inter-compatible with
each other leading to a clear difference in motivation, desire and reason to participate. This
observation is brought up in literature as well as by interview respondents as seen in section
5.2. The question arises if these two focuses could after all be combined and would it
improve the hackathon event outcome sustainability?
As noted in section 4.4 remote participation is mainly viewed negatively based on the topic
related publications. E.g., Some researchers suggest that outcome sustainability is not worth
investing in while others cautiously believe in the possibility of doing so. Although very
little studied, I will personally have to agree with this negative attitude based on my own
first-hand experience. Participating in several events during studies as an attendee as well as
working at DigiEduHack 2020 as a facilitator has been a valuable experience for
understanding remote participation methodology with all of its pros and cons. It does feel
less practical, intuitive and “real” for lack of a better word. One of the respondents (R3)
mentioned in the interview (Table 12) something similar, stating that “the feeling is not
there” in remotely held events. On the other hand, it opens up a lot of before-impossible
possibilities such as international collaboration, a larger pool of potential participants and of
course lower threshold to participate. I just can’t help but wonder if the above motioned
negative attitudes originate from unfamiliarity and relative newness to remote participation
framework. It must be pretty challenging to get used to this approach after doing it physically
and face-to-face for entire life. Technology is also progressing in an extraordinary phase
leading to a belief that current drawback and cons will be eliminated. The technological
solutions of virtual- and augmented reality might be a game-changer in the future when it
comes to remote participating in hackathon like events.
Considering hackathon event outcome sustainability, organizers must think of how these
events should be organized. E.g., Have educational units measured the effect of Problem-
and Challenge Based Learning in education , and how this methodology affects
students results. In case results are positive, it might be reasonable to try to integrate this
methodology more into the hackathons as well. Understanding the underlying motivation to
participate in hackathon like event is really important as well. Why do people participate,
collaborate  and co-operate in these events ? Why are participants willing to give
their own time and resources for greater good , ? Also, it seems like the literature
doesn’t provide an answer to what is combined result of different motivators? What are
participants, organizers, challenge owners and other stakeholders’ true motivations to
engage in hackathon like events and how it correlates with event outcome sustainability?
This study has been focusing on literature review related to the subject of hackathon like
events with a special focus on the following research question:
RQ1) Status of post-hackathon outcomes sustainability
RQ2) Remote approach as an option for participation in hackathons
Systematic Literature Review (SLR) was used as a primary method for collecting and
selecting related publications for this thesis work. During the in-depth literature analysis, the
following main observations were made:
1) Publications on post-event walkthrough are the most common ones amongst
collected literature. These publications explain how the event was organised with
2) There is a decent amount of research on motivation during the event, although it
mainly focuses on participant motivators
3) Hackathons are commonly viewed as part of the learning process that in many cases
have similarities with PBL and/or emphasise the importance of teamwork
4) Event outcome sustainability (RQ1) is little researched and has contradictory
interpretations, ranging from extremely pessimistic to somewhat promising ones
5) Remote participation in hackathon like events (RQ2) is almost not studied and mostly
limited to subjective sidenotes in research focusing on other subjects
Expert interviews were conducted to compare literature findings with real-life experiences
and possibly get more insight specifically on observations 4 and 5 from the list above.
Respondents seemed not to be too worried about hackathon outcome sustainability (RQ1)
but rather highlighted that it should be planned in the event in advance. As of remote
participation in hackathons (RQ2), respondents identified clear advantages like practicality
of online attendance and opportunity for more international events, as well as disadvantages
such as requiring more mentoring recourses and lack of notorious hackathon affiliated
 T. Lodato and C. Disalvo, ‘Issue-oriented hackathons as material participation’, New
Media Soc., vol. 18, Feb. 2016, doi: 10.1177/1461444816629467.
 G. Briscoe and C. Mulligan, ‘Digital Innovation: The Hackathon Phenomenon’,
undefined, 2014. /paper/Digital-Innovation%3A-The-Hackathon-Phenomenon-
Briscoe/cb8e44ec1bcd6062e5fccafb6837030be334731d (accessed Nov. 19, 2020).
 ‘OpenBSD: Hackathons’. https://www.openbsd.org/hackathons.html (accessed Nov. 19,
 E. P. P. Pe-Than, A. Nolte, A. Filippova, C. Bird, S. Scallen, and J. D. Herbsleb,
‘Designing Corporate Hackathons With a Purpose: The Future of Software
Development’, IEEE Softw., vol. 36, no. 1, pp. 15–22, Jan. 2019, doi:
 E. H. Trainer, A. Kalyanasundaram, C. Chaihirunkarn, and J. D. Herbsleb, ‘How to
Hackathon: Socio-technical Tradeoffs in Brief, Intensive Collocation’, in Proceedings
of the 19th ACM Conference on Computer-Supported Cooperative Work & Social
Computing, New York, NY, USA, Feb. 2016, pp. 1118–1130, doi:
 A. Filippova, E. Trainer, and J. D. Herbsleb, ‘From Diversity by Numbers to Diversity
as Process: Supporting Inclusiveness in Software Development Teams with
Brainstorming’, in 2017 IEEE/ACM 39th International Conference on Software
Engineering (ICSE), Buenos Aires, May 2017, pp. 152–163, doi:
 A. Nolte, E. P. P. Pe-Than, A. Filippova, C. Bird, S. Scallen, and J. D. Herbsleb, ‘You
Hacked and Now What? - Exploring Outcomes of a Corporate Hackathon’, Proc. ACM
Hum.-Comput. Interact., vol. 2, no. CSCW, p. 129:1–129:23, Nov. 2018, doi:
 P. Johnson and P. Robinson, ‘Civic Hackathons: Innovation, Procurement, or Civic
Engagement?’, Rev. Policy Res., vol. 31, no. 4, pp. 349–357, 2014, doi:
 J. Wilson, K. Bender, and J. DeChants, ‘Beyond the Classroom: The Impact of a
University-Based Civic Hackathon Addressing Homelessness’, J. Soc. Work Educ., vol.
55, no. 4, pp. 736–749, Oct. 2019, doi: 10.1080/10437797.2019.1633975.
 D. Cobham, C. Gowen, K. Jacques, J. Laurel, and S. Ringham, ‘FROM APPFEST
TO ENTREPRENEURS: USING A HACKATHON EVENT TO SEED A
UNIVERSITY STUDENT- LED ENTERPRISE’, p. 8.
 J. R. Byrne, K. O’Sullivan, and K. Sullivan, ‘An IoT and Wearable Technology
Hackathon for Promoting Careers in Computer Science’, IEEE Trans. Educ., vol. 60,
no. 1, pp. 50–58, Feb. 2017, doi: 10.1109/TE.2016.2626252.
 J. K. Silver, D. S. Binder, N. Zubcevik, and R. D. Zafonte, ‘Healthcare Hackathons
Provide Educational and Innovation Opportunities: A Case Study and Best Practice
Recommendations’, J. Med. Syst., vol. 40, 2016, doi: 10.1007/s10916-016-0532-3.
 K. R. Olson et al., ‘Health hackathons: theatre or substance? A survey assessment of
outcomes from healthcare-focused hackathons in three countries’, BMJ Innov., vol. 3,
no. 1, pp. 37–44, Feb. 2017, doi: 10.1136/bmjinnov-2016-000147.
 A. Happonen, D. Minashkina, A. Nolte, and M. A. M. Angarita, ‘Hackathons as a
company – University collaboration tool to boost circularity innovations and
digitalization enhanced sustainability’, AIP Conf. Proc., vol. 2233, no. 1, p. 050009,
May 2020, doi: 10.1063/5.0001883.
 A. Nandi and M. Mandernach, ‘Hackathons as an Informal Learning Platform’, in
Proceedings of the 47th ACM Technical Symposium on Computing Science Education,
New York, NY, USA, Feb. 2016, pp. 346–351, doi: 10.1145/2839509.2844590.
 A. Richterich, ‘Hacking events: Project development practices and technology use at
hackathons’:, Convergence, May 2017, doi: 10.1177/1354856517709405.
 J. Warner and P. J. Guo, ‘Hack.edu: Examining How College Hackathons Are
Perceived By Student Attendees and Non-Attendees’, in Proceedings of the 2017 ACM
Conference on International Computing Education Research, New York, NY, USA,
Aug. 2017, pp. 254–262, doi: 10.1145/3105726.3106174.
 W. Uys, ‘Hackathons as a Formal Teaching Approach in Information Systems
Capstone Courses’, 2020, pp. 79–95.
 M. Komssi, D. Pichlis, M. Raatikainen, K. Kindström, and J. Järvinen, ‘What are
Hackathons for?’, IEEE Softw., vol. 32, no. 5, pp. 60–67, Sep. 2015, doi:
 M. Komssi, V. dal Blanco, K. Kindström, J. Järvinen, and M. Raatikainen, ‘Industrial
Experiences of Organizing a Hackathon to Assess a Device-centric Cloud Ecosystem’,
article/compsac/2013/4986a790/12OmNyoiYYl (accessed Nov. 19, 2020).
 A. Decker, K. Eiselt, and K. Voll, ‘Understanding and improving the culture of
hackathons: Think global hack local’, in 2015 IEEE Frontiers in Education Conference
(FIE), Oct. 2015, pp. 1–8, doi: 10.1109/FIE.2015.7344211.
 B. Rosell, S. Kumar, and J. Shepherd, ‘Unleashing innovation through internal
hackathons’, in 2014 IEEE Innovations in Technology Conference, May 2014, pp. 1–8,
 H. Abdullah and J. Mtsweni, ‘Stimulating and maintaining students’ interest in
computer science using the hackathon model’, Indep. J. Teach. Learn., Jan. 2015.
 J. Porras, A. Knutas, J. Ikonen, A. Happonen, J. Khakurel, and A. Herala, ‘Code
camps and hackathons in education - literature review and lessons learned’, Jan. 2019,
 C. Ebert, P. Abrahamsson, and N. Oza, ‘Lean Software Development’, Softw. IEEE,
vol. 29, pp. 22–25, Sep. 2012, doi: 10.1109/MS.2012.116.
 A. Happonen and D. Minashkina, ‘The Code Camp Story: Facilitators and
Participants View into University – Company Collaboration’, TYYLI -
TYÖELÄMÄJAKSOJA JA TYÖSSÄOPPIMISTA YLIOPISTO-OPINTOIHIN, Apr. 12,
and-participants-view-into-university-company-collaboration/ (accessed Nov. 27,
 A. Happonen, U. Santti, and H. Auvinen, ‘Digitalization Boosted Recycling:
Gamification as an Inspiration for Young Adults to do Enhanced Waste Sorting’, Nov.
2019, vol. 2233, doi: 10.1063/5.0001547.
 J. Ikonen, A. Happonen, and J. Porras, ‘Game Programming for Learning -
Experiments in Teaching Protocols with Game Programming.’, Jul. 2007, doi:
 A. Happonen and D. Minashkina, ‘Ideas and experiences from university : industry
collaboration : hackathons, code camps and citizen participation’, Lappeenranta
University of Technology, report, 2018. Accessed: Nov. 27, 2020. [Online]. Available:
 I. C. Florea, O. A. Vochin, L. Ciachir, and P. Nagel-Picioruş, ‘COMPETITION FOR
INNOVATION IN THE FINANCIAL SOFTWARE INDUSTRY – A RESEARCH ON
HACKATHONS’, p. 9.
 M. Flores et al., ‘How Can Hackathons Accelerate Corporate Innovation?’, 2018, pp.
 M. D. Sakhumuzi and O. K. Emmanuel, ‘Student perception of the contribution of
Hackathon and collaborative learning approach on computer programming pass rate’, in
2017 Conference on Information Communication Technology and Society (ICTAS),
Mar. 2017, pp. 1–5, doi: 10.1109/ICTAS.2017.7920524.
 G. Briscoe, T. E. Virani, and M. Dima, ‘Hackathons: Why Co-Location?’, Jan. 2015,
Accessed: Nov. 20, 2020. [Online]. Available:
 K. Gama, B. Alencar, F. Calegario, A. Neves, and P. Alessio, ‘A Hackathon
Methodology for Undergraduate Course Projects’, Oct. 2018, pp. 1–9, doi:
 E. Salmela, A. Happonen, M. Hirvimäki, and I. Vimm, ‘Is Time Pressure an
Advantage or a Disadvantage for Front End Innovation – Case Digital Jewelry’, J. Innov.
Manag., vol. 3, no. 4, Art. no. 4, 2015, doi: 10.24840/2183-0606_003.004_0005.
 S. Sousa, ‘Lessons learnt from a publicprivate big data hackathon’, The British
Library. Accessed: Nov. 20, 2020. [Online]. Available: https://www.bl.uk/collection-
 K. Gama, ‘Crowdsourced Software Development in Civic Apps - Motivations of
Civic Hackathons Participants’, Jan. 2017, pp. 550–555, doi:
 L. Chrisopherson and S. Ahalt, ‘Developing Scientific Software through the Open
Community Engagement Process’, Nov. 2013, doi: 10.6084/m9.figshare.790723.
 M. Gotta and K. Moyer, ‘The Hackathon Guide for Digital Workplace Leaders’,
digital-workplace-leaders (accessed Nov. 20, 2020).
 F. Kitsios and M. Kamariotou, ‘Beyond Open Data Hackathons: Exploring Digital
Innovation Success’, Information, vol. 10, no. 7, Art. no. 7, Jul. 2019, doi:
 H. Kienzler and C. Fontanesi, ‘Learning through inquiry: a Global Health
Hackathon’, Teach. High. Educ., vol. 22, no. 2, pp. 129–142, Feb. 2017, doi:
 S. Hartmann, A. Mainka, and W. Stock, ‘Innovation Contests: How to Engage
Citizens in Solving Urban Problems?’, 2018, pp. 254–273.
 S. Henderson, ‘Getting the Most Out of Hackathons for Social Good’, 2015, pp. 182–
 M. Drouhard, A. Tanweer, and B. Fiore-Gartland, ‘Typology of Hackathon Events’,
2017, Accessed: Nov. 23, 2020. [Online]. Available:
 J. Khakurel, J. Porras, A. Knutas, A. Herala, J. Ikonen, and A. Happonen,
‘Hackathons in Software Engineering Education – Lessons Learned from a Decade of
Events’, Mar. 2018, doi: 10.1145/3194779.3194783.
 G. Briscoe and J. Hon, ‘Choreograthons: Hackathons for Dance’, undefined, 2016.
Hon/90c8a31e9ffd51aa068ed952f6a4136bb7ba3438 (accessed Nov. 20, 2020).
 A. Nolte, L. B. Hayden, and J. D. Herbsleb, ‘How to Support Newcomers in
Scientific Hackathons - An Action Research Study on Expert Mentoring’, Proc. ACM
Hum.-Comput. Interact., vol. 4, no. CSCW1, p. 025:1–025:23, May 2020, doi:
 E. Porter, C. Bopp, E. Gerber, and A. Voida, ‘Reappropriating Hackathons: The
Production Work of the CHI4Good Day of Service’, in Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems, New York, NY, USA, May 2017,
pp. 810–814, doi: 10.1145/3025453.3025637.
 J. L. Zapico Lamela, D. Pargman, H. Ebner, and E. Eriksson, ‘Hacking
sustainability : Broadening participation through Green Hackathons’, presented at the
Fourth International Symposium on End-User Development. June 10-13, 2013, IT
University of Copenhagen, Denmark, 2013, Accessed: Nov. 20, 2020. [Online].
 A. Fowler, ‘Informal STEM Learning in Game Jams, Hackathons and Game
Creation Events’, in Proceedings of the International Conference on Game Jams,
Hackathons, and Game Creation Events, New York, NY, USA, Mar. 2016, pp. 38–41,
 M. Lara and K. Lockwood, ‘Hackathons as Community-Based Learning: a Case
Study’, TechTrends, vol. 60, no. 5, pp. 486–495, Sep. 2016, doi: 10.1007/s11528-016-
 R. Cameron Craddock et al., ‘Brainhack: a collaborative workshop for the open
neuroscience community’, GigaScience, vol. 5, p. 16, 2016, doi: 10.1186/s13742-016-
 P. Angelidis et al., ‘The hackathon model to spur innovation around global mHealth’,
J. Med. Eng. Technol., vol. 40, no. 7–8, pp. 392–399, Nov. 2016, doi:
 S. Möller et al., ‘Community-driven development for computational biology at
Sprints, Hackathons and Codefests’, BMC Bioinformatics, vol. 15, no. 14, p. S7, Nov.
2014, doi: 10.1186/1471-2105-15-S14-S7.
 E. H. Trainer, A. Kalyanasundaram, and J. D. Herbsleb, ‘E-Mentoring for Software
Engineering: a Socio-Technical Perspective’, in 2017 IEEE/ACM 39th International
Conference on Software Engineering: Software Engineering Education and Training
Track (ICSE-SEET), May 2017, pp. 107–116, doi: 10.1109/ICSE-SEET.2017.19.
 K. Gama, ‘Developing Course Projects in a Hack Day: An Experience Report’, in
Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer
Science Education, New York, NY, USA, Jul. 2019, pp. 388–394, doi:
 F. Senghore, E. Campos-Nanez, P. Fomin, and J. S. Wasek, ‘Applying social network
analysis to validate mass collaboration innovation drivers: An empirical study of
NASA’s International Space Apps Challenge’, J. Eng. Technol. Manag., vol. 37, pp. 21–
31, Jul. 2015, doi: 10.1016/j.jengtecman.2015.08.007.
 N. Tsjardiwal, ‘Hackathons: An Effective Communication Tool for Innovation
within an Organization?’, undefined, 2016. /paper/Hackathons%3A-An-Effective-
Tsjardiwal/360b1ebef9c6c75aafa20b6b7831a3aeb82a55b3 (accessed Nov. 20, 2020).
 F. Page, S. Sweeney, F. Bruce, and S. Baxter, ‘The use of the “hackathon” in design
education: An opportunistic exploration’, in Proceedings of the 18th International
Conference on Engineering and Product Design Education: Design Education:
Collaboration and Cross-Disciplinarity, E and PDE 2016, 2016, pp. 246–251,
Accessed: Nov. 20, 2020. [Online]. Available:
 C. Jennett, S. Papadopoulou, J. Himmelstein, A. Vaugoux, V. Roger, and A. L. Cox,
‘Case Study 3: Students’ Experiences of Interdisciplinary Learning while Building
Scientific Video Games’, Int. J. Game-Based Learn. IJGBL, vol. 7, no. 3, pp. 93–97,
2017, doi: 10.4018/IJGBL.2017070110.
 E. Morozov, To Save Everything, Click Here: The Folly of Technological
Solutionism. PublicAffairs, 2013.
 C. Rennick, C. Hulls, D. Wright, A. Milne, E. Li, and S. Bedi, ‘Engineering Design
Days: Engaging Students with Authentic Problem-Solving in an Academic Hackathon’,
2018, doi: 10.18260/1-2--30407.
 L. Irani, ‘Hackathons and the Making of Entrepreneurial Citizenship’:, Sci. Technol.
Hum. Values, Apr. 2015, doi: 10.1177/0162243915578486.
 J. Falk Olesen and K. Halskov, ‘10 Years of Research With and On Hackathons’, in
Proceedings of the 2020 ACM Designing Interactive Systems Conference, New York,
NY, USA, Jul. 2020, pp. 1073–1088, doi: 10.1145/3357236.3395543.
 B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman,
‘Systematic literature reviews in software engineering – A systematic literature review’,
Inf. Softw. Technol., vol. 51, no. 1, pp. 7–15, Jan. 2009, doi:
 R. Armstrong, B. J. Hall, J. Doyle, and E. Waters, ‘“Scoping the scope” of a cochrane
review’, J. Public Health, vol. 33, no. 1, pp. 147–150, Mar. 2011, doi:
 M. Q. Patton, Qualitative Research & Evaluation Methods. SAGE, 2002.
 Elsevier, ‘About Scopus - Abstract and citation database | Elsevier’, Elsevier.com.
https://www.elsevier.com/solutions/scopus (accessed Nov. 10, 2020).
 P. A. Horton, S. S. Jordan, S. Weiner, and M. Lande, ‘Project-Based Learning
Among Engineering Students During Short-Form Hackathon Events’, presented at the
2018 ASEE Annual Conference & Exposition, Jun. 2018, Accessed: Nov. 21, 2020.
[Online]. Available: https://peer.asee.org/project-based-learning-among-engineering-
 C. L. Place, S. S. Jordan, M. Lande, and S. Weiner, ‘Engineering Students Rapidly
Learning at Hackathon Events’, presented at the 2017 ASEE Annual Conference &
Exposition, Jun. 2017, Accessed: Nov. 21, 2020. [Online]. Available:
 I.-A. Chounta, S. Manske, and H. U. Hoppe, ‘“From Making to Learning”:
introducing Dev Camps as an educational paradigm for Re-inventing Problem-based
Learning’, Int. J. Educ. Technol. High. Educ., vol. 14, no. 1, p. 21, Sep. 2017, doi:
 C. Anslow, J. Brosz, F. Maurer, and M. Boyes, ‘Datathons: An Experience Report
of Data Hackathons for Data Science Education’, Feb. 2016, pp. 615–620, doi:
 P. C. Blumenfeld, E. Soloway, R. W. Marx, J. S. Krajcik, M. Guzdial, and A.
Palincsar, ‘Motivating Project-Based Learning: Sustaining the Doing, Supporting the
Learning’, Educ. Psychol., vol. 26, no. 3–4, pp. 369–398, Jun. 1991, doi:
 M. Nichols, K. Cator, and M. Torres, ‘Challenge Based Learning Guide’, p. 59, 2016.
 D. F. Wood, ‘Problem based learning’, BMJ, vol. 336, no. 7651, pp. 971–971, May
2008, doi: 10.1136/bmj.39546.716053.80.
 H. Beetham and R. Sharpe, ‘Rethinking Pedagogy for a Digital Age: Designing and
Delivering E-Learning’, undefined, 2007. /paper/Rethinking-Pedagogy-for-a-Digital-
(accessed Nov. 21, 2020).
 A. Bruns, ‘Produsage’, in Proceedings of the 6th ACM SIGCHI conference on
Creativity & cognition, New York, NY, USA, Jun. 2007, pp. 99–106, doi:
 L. W. Anderson, ‘Taxonomies of Educational Objectives as Bases for Curriculum
Planning’, Oxford Research Encyclopedia of Education, 2018.
ore-9780190264093-e-1038 (accessed Nov. 21, 2020).
 J. Fransen, P. A. Kirschner, and G. Erkens, ‘Mediating team effectiveness in the
context of collaborative learning: The importance of team and task awareness’, Comput.
Hum. Behav., vol. 27, no. 3, pp. 1103–1113, May 2011, doi: 10.1016/j.chb.2010.05.017.
 F. M. de T. Piza et al., ‘Assessing team effectiveness and affective learning in a
datathon’, Int. J. Med. Inf., vol. 112, pp. 40–44, Apr. 2018, doi:
 S. Sarin and C. McDermott, ‘The Effect of Team Leader Characteristics on Learning,
Knowledge Application, and Performance of Cross-Functional New Product
Development Teams’, Decis. Sci., vol. 34, pp. 707–739, Dec. 2003, doi: 10.1111/j.1540-
 ‘A study of the design process’. Design Council. London, UK, Accessed: Nov. 21,
2020. [Online]. Available:
 S. J. Liebowitz and K. P. de Meuse, ‘The Application of Team Building’:, Hum.
Relat., Apr. 2016, doi: 10.1177/001872678203500102.
 S. I. Tannenbaum, R. L. Beard, and E. Salas, ‘Chapter 5 Team Building and its
Influence on Team Effectiveness: an Examination of Conceptual and Empirical
Developments’, in Advances in Psychology, vol. 82, K. Kelley, Ed. North-Holland,
1992, pp. 117–153.
 K. P. D. Meuse, ‘Teamwork: What Must Go Right/What Can Go Wrong Carl E.
Larson and Frank M. J. LaFasto, Sage Publications, Newbury Park, California, 1989,
No. of Pages: 150’, J. Organ. Behav., vol. 13, no. 6, pp. 637–638, 1992, doi:
 D. V. Day, P. Gronn, and E. Salas, ‘Leadership capacity in teams’, Leadersh. Q., vol.
15, no. 6, pp. 857–880, Dec. 2004, doi: 10.1016/j.leaqua.2004.09.001.
 S. J. Zaccaro, B. Heinen, and M. Shuffler, ‘Team leadership and team effectiveness’,
in Team effectiveness in complex organizations: Cross-disciplinary perspectives and
approaches, New York, NY, US: Routledge/Taylor & Francis Group, 2009, pp. 83–111.
 S. J. Zaccaro, A. L. Rittman, and M. A. Marks, ‘Team leadership’, Leadersh. Q., vol.
12, no. 4, pp. 451–483, Dec. 2001, doi: 10.1016/S1048-9843(01)00093-5.
 C. L. Pearce and H. P. Sims Jr., ‘Vertical versus shared leadership as predictors of
the effectiveness of change management teams: An examination of aversive, directive,
transactional, transformational, and empowering leader behaviors’, Group Dyn. Theory
Res. Pract., vol. 6, no. 2, pp. 172–197, 2002, doi: 10.1037/1089-26220.127.116.11.
 D. Cobham, C. Gowen, B. Hargrave, K. Jacques, J. Laurel, and S. Ringham, ‘FROM
HACKATHON TO STUDENT ENTERPRISE: AN EVALUATION OF CREATING
SUCCESSFUL AND SUSTAINABLE STUDENT ENTREPRENEURIAL ACTIVITY
INITIATED BY A UNIVERSITY HACKATHON’, EDULEARN17 Proc., pp. 789–
 R. A. Heifetz, M. Linsky, and A. Grashow, The Practice of Adaptive Leadership:
Tools and Tactics for Changing Your Organization and the World, 1st edition. Boston,
Mass: Harvard Business Press, 2009.
 P. Niemelä, T. Partanen, T. Toivanen, T. Toikkanen, V. Kangas, and M. Översti,
‘Code ABC Hackathons: Teachers as Tinkerers’, in Digital Turn in Schools—Research,
Policy, Practice, Singapore, 2019, pp. 157–169, doi: 10.1007/978-981-13-7361-9_11.
 C. N. Quinn, ‘Pragmatic Evaluation: Lessons from Usability’, 1996, Accessed: Nov.
21, 2020. [Online]. Available:
 G. Juell-Skielse, A. Hjalmarsson, P. Johannesson, and D. Rudmark, ‘Is the Public
Motivated to Engage in Open Data Innovation?’, in Electronic Government, Berlin,
Heidelberg, 2014, pp. 277–288, doi: 10.1007/978-3-662-44426-9_23.
 A. Purwanto, A. Zuiderwijk, and M. Janssen, ‘Citizens’ Motivations for Engaging in
Open Data Hackathons’, in Electronic Participation, Cham, 2019, pp. 130–141, doi:
 A. Simonofski, V. Amaral de Sousa, A. Clarinval, and B. Vanderose, ‘Participation
in Hackathons: A Multi-methods View on Motivators, Demotivators and Citizen
Participation’, in Research Challenges in Information Science, Cham, 2020, pp. 229–
246, doi: 10.1007/978-3-030-50316-1_14.
 C. Guerrero, M. del M. Leza, Y. González, and A. Jaume-i-Capó, ‘Analysis of the
results of a hackathon in the context of service-learning involving students and
professionals’, in 2016 International Symposium on Computers in Education (SIIE),
Sep. 2016, pp. 1–6, doi: 10.1109/SIIE.2016.7751857.
 A. Zuiderwijk, M. Janssen, and Y. K. Dwivedi, ‘Acceptance and use predictors of
open data technologies: Drawing upon the unified theory of acceptance and use of
technology’, Gov. Inf. Q., vol. 32, no. 4, pp. 429–440, Oct. 2015, doi:
 M. Calco and A. Veeck, ‘The Markathon: Adapting the Hackathon Model for an
Introductory Marketing Class Project’, Mark. Educ. Rev., vol. 25, no. 1, pp. 33–38, Jan.
2015, doi: 10.1080/10528008.2015.999600.
 V. Weerakkody, Z. Irani, K. Kapoor, U. Sivarajah, and Y. K. Dwivedi, ‘Open data
and its usability: an empirical view from the Citizen’s perspective’, Inf. Syst. Front., vol.
19, no. 2, pp. 285–300, Apr. 2017, doi: 10.1007/s10796-016-9679-1.
 V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, ‘User Acceptance of
Information Technology: Toward a Unified View’, MIS Q., vol. 27, no. 3, pp. 425–478,
2003, doi: 10.2307/30036540.
 A. Purwanto, A. Zuiderwijk, and M. Janssen, ‘Citizen engagement in an open
election data initiative: a case study of Indonesian’s “Kawal Pemilu”’, in Proceedings
of the 19th Annual International Conference on Digital Government Research:
Governance in the Data Age, New York, NY, USA, May 2018, pp. 1–10, doi:
 M. Meriläinen and R. Aurava, ‘Internal Barriers to Entry for First-Time Participants
in the Global Game Jam’, Oct. 2018.
 A. Arya, J. Chastine, J. Preston, and A. Fowler, ‘An International Study on Learning
and Process Choices in the Global Game Jam’, Int. J. Game-Based Learn. IJGBL, vol.
3, no. 4, pp. 27–46, 2013, doi: 10.4018/ijgbl.2013100103.
 A. C. Petri, C. Schindler, W. Slany, B. Spieler, and J. Smith, ‘Pocket Game Jams: a
Constructionist Approach at Schools’, in MobileHCI ’15 Proceedings of the 17th
International Conference on Human-Computer Interaction with Mobile Devices and
Services Adjunct, 2015, pp. 156–526, doi: 10.1145/2786567.2801610.
 N. Wearn and B. McDonald, ‘Ethos of location and its implication to the motivators
of Global Games Jam participants’, in Proceedings of the International Conference on
Game Jams, Hackathons, and Game Creation Events, New York, NY, USA, Mar. 2016,
pp. 58–61, doi: 10.1145/2897167.2897176.
 E. H. Trainer, C. Chaihirunkarn, A. Kalyanasundaram, and J. D. Herbsleb,
‘Community Code Engagements: Summer of Code & Hackathons for Community
Building in Scientific Software’, in Proceedings of the 18th International Conference
on Supporting Group Work, New York, NY, USA, Nov. 2014, pp. 111–121, doi:
 E. Trainer, C. Chaihirunkarn, and J. Herbsleb, ‘The Big Effects of Short-term Efforts:
A Catalyst for Community Engagement in Scientific Software’, Nov. 2013, doi:
 R. E. Kraut et al., Building Successful Online Communities: Evidence-Based Social
Design. Cambridge, Mass: The MIT Press, 2012.
 S. Worchel, H. Rothgerber, E. A. Day, D. Hart, and J. Butemeyer, ‘Social identity
and individual productivity within groups’, Br. J. Soc. Psychol., vol. 37 ( Pt 4), pp. 389–
413, Dec. 1998, doi: 10.1111/j.2044-8309.1998.tb01181.x.
 S. Gaertner, J. Dovidio, B. Banker, M. Houlette, K. Johnson, and E. McGlynn,
‘Reducing intergroup conflict: From superordinate goals to decategorization,
recategorization, and mutual differentiation’, Group Dyn. Theory Res. Pract., vol. 4, pp.
98–114, Mar. 2000, doi: 10.1037/1089-2618.104.22.168.
 P. Shannon et al., ‘Cytoscape: A Software Environment for Integrated Models of
Biomolecular Interaction Networks’, Genome Res., vol. 13, no. 11, pp. 2498–2504, Nov.
2003, doi: 10.1101/gr.1239303.
 A. R. Pico, T. Kelder, M. P. van Iersel, K. Hanspers, B. R. Conklin, and C. Evelo,
‘WikiPathways: Pathway Editing for the People’, PLOS Biol., vol. 6, no. 7, p. e184, Jul.
2008, doi: 10.1371/journal.pbio.0060184.
 K. Gama, B. Alencar Gonçalves, and P. Alessio, ‘Hackathons in the formal learning
process’, in Proceedings of the 23rd Annual ACM Conference on Innovation and
Technology in Computer Science Education, New York, NY, USA, Jul. 2018, pp. 248–
253, doi: 10.1145/3197091.3197138.
 M. A. Medina Angarita and A. Nolte, ‘What Do We Know About Hackathon
Outcomes and How to Support Them? – A Systematic Literature Review’, in
Collaboration Technologies and Social Computing, Cham, 2020, pp. 50–64, doi:
 A. Mantzavinou, B. J. Ranger, S. Gudapakkam, K. G. Broach Hutchins, E. Bailey,
and K. R. Olson, ‘Health Hackathons Drive Affordable Medical Technology Innovation
Through Community Engagement’, in Technologies for Development, Cham, 2018, pp.
87–95, doi: 10.1007/978-3-319-91068-0_8.
 J. S. Bell, F. E. Murray, and E. L. Davies, ‘An investigation of the features facilitating
effective collaboration between public health experts and data scientists at a hackathon’,
Public Health, vol. 173, pp. 120–125, Aug. 2019, doi: 10.1016/j.puhe.2019.05.007.
 F. Kitsios and M. Kamariotou, ‘Open data hackathons: an innovative strategy to
enhance entrepreneurial intention’, Int. J. Innov. Sci., vol. 10, May 2018, doi:
 M. Kamariotou and F. Kitsios, ‘Open Data Hackathons: A Strategy to Increase
Innovation in the City’, Aug. 2017.
 C. H. Brooks, ‘Community connections: lessons learned developing and maintaining
a computer science service-learning program’, ACM SIGCSE Bull., vol. 40, no. 1, pp.
352–356, Mar. 2008, doi: 10.1145/1352322.1352256.
 M. Alba, M. Avalos, C. Guzmán, and V. M. Larios, ‘Synergy between smart cities’
hackathons and living labs as a vehicle for accelerating tangible innovations on cities’,
in 2016 IEEE International Smart Cities Conference (ISC2), Sep. 2016, pp. 1–6, doi:
 M. Riel, J. Rhoads, E. Ellis, M. Riel, J. Rhoads, and E. Ellis, ‘Culture of Critique:
Online Learning Circles and Peer Reviews in Graduate Education’, http://services.igi-
global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-59140-965-6.ch006, Jan. 01,
1AD. https://www.igi-global.com/gateway/chapter/28801 (accessed Nov. 22, 2020).
 T. Chan, J. McMurray, A. Levy, H. Sveistrup, and J. Wallace, ‘Post-Hackathon
Learning Circles: Supporting Lean Startup Development’, in Extended Abstracts of the
2020 CHI Conference on Human Factors in Computing Systems, New York, NY, USA,
Apr. 2020, pp. 1–8, doi: 10.1145/3334480.3375216.
 J. McBride and J. Good, ‘Learning circles (CDI): What is their potential Aboriginal
community economic development?’, Feb. 02, 2017.
community-economic-development (accessed Nov. 22, 2020).
 A. Sadovykh et al., ‘On the Use of Hackathons to Enhance Collaboration in Large
Collaborative Projects : - A Preliminary Case Study of the MegaM@Rt2 EU Project -’,
in 2019 Design, Automation Test in Europe Conference Exhibition (DATE), Mar. 2019,
pp. 498–503, doi: 10.23919/DATE.2019.8715247.
 N. Morgan, ‘5 Fatal Flaws With Virtual Meetings’, Forbes.
meetings/ (accessed Nov. 23, 2020).
 S. Teasley, L. Covi, M. S. Krishnan, and J. S. Olson, ‘How does radical collocation
help a team succeed?’, in Proceedings of the 2000 ACM conference on Computer
supported cooperative work, New York, NY, USA, Dec. 2000, pp. 339–346, doi:
 E. P. P. Pe-Than and J. D. Herbsleb, ‘Understanding Hackathons for Science:
Collaboration, Affordances, and Outcomes’, in Information in Contemporary Society,
Cham, 2019, pp. 27–37, doi: 10.1007/978-3-030-15742-5_3.
 M. Palacin-Silva, J. Khakurel, A. Happonen, T. Hynninen, and J. Porras, ‘Infusing
Design Thinking into a Software Engineering Capstone Course’, in 2017 IEEE 30th
Conference on Software Engineering Education and Training (CSEE T), Nov. 2017, pp.
212–221, doi: 10.1109/CSEET.2017.41.
 A. Happonen et al., ‘Art-technology Collaboration and Motivation Sources in
Technologically Supported Artwork Buildup Project’, Phys. Procedia, vol. 78, pp. 407–
414, Jan. 2015, doi: 10.1016/j.phpro.2015.11.055.
 A. Happonen and vesa siljander, ‘Gainsharing in Logistics Outsourcing: Trust leads
to Success in the Digital Era’, Int. J. Collab. Enterp., vol. 6, pp. 150–175, Sep. 2020,
 V. Palacin et al., ‘SENSEI: Harnessing Community Wisdom for Local
Environmental Monitoring in Finland’, in Extended Abstracts of the 2019 CHI
Conference on Human Factors in Computing Systems, New York, NY, USA, May 2019,
pp. 1–8, doi: 10.1145/3290607.3299047.
 S. Orchard, A. Happonen, V. Palacin, S. Gilbert, A. Eaton, and F. Angela, ‘Drivers
of Participation in Digital Citizen Science: Case Studies on Järviwiki and Safecast’,
Citiz. Sci. Theory Pract., vol. 5, pp. 1–20, Oct. 2020, doi: 10.5334/cstp.290.