Crowdsourcing in Design Research - Potentials & Limitations

Abstract

Crowdsourcing has by now proven itself particularly useful in the early phases of product design, outperforming traditional in-house R&D. However, little attention has been given to the application of crowdsourcing in design research where the cost and time of planning and executing studies, combined with inherent biases in selection of participants, unintentional priming of and inadvertent influence of subjects responses can ruin the outcome. Additionally, when unexpected and unexplainable findings occur, the value and validity of the study can be seriously undermined. Here lean, quick crowdsourcing experiments could aid in planning of conventional research or in some cases replacing these completely. We propose a Six Step Co-creation Cycle method and apply it to thirty projects with varying level of innovativeness. In doing so we uncover a range of pros, such as uncovering biases, collecting new knowledge and building a research network and cons, such as limited ability to synthesize ideas and the need to eliminating noise, of applying crowdsourcing in the early phases of design research.

Full text

ICED13/164 1
INTERNATIONAL CONFERENCE ON ENGINEERING DESIGN, ICED13
19-22 AUGUST 2013, SUNGKYUNKWAN UNIVERSITY, SEOUL, KOREA
CROWDSOURCING IN DESIGN RESEARCH -
POTENTIALS & LIMITATIONS
Soren Ingomar PETERSEN
ingomar&ingomar - consulting, United States of America
ABSTRACT
Crowdsourcing has by now proven itself particularly useful in the early phases of product design,
outperforming traditional in-house R&D. However, little attention has been given to the application of
crowdsourcing in design research where the cost and time of planning and executing studies,
combined with inherent biases in selection of participants, unintentional priming of and inadvertent
influence of subjects responses can ruin the outcome. Additionally, when unexpected and
unexplainable findings occur, the value and validity of the study can be seriously undermined. Here
lean, quick crowdsourcing experiments could aid in planning of conventional research or in some
cases replacing these completely. We propose a Six Step Co-creation Cycle method and apply it to
thirty projects with varying level of innovativeness. In doing so we uncover a range of pros, such as
uncovering biases, collecting new knowledge and building a research network and cons, such as
limited ability to synthesize ideas and the need to eliminating noise, of applying crowdsourcing in the
early phases of design research.
Keywords: crowdsourcing, social networks, new social media, six step co-creation cycle, wisdom of
crowds, design research
Contact:
Dr. Soren Ingomar Petersen
ingomar&ingomar - consulting
design quantification
South Pasadena
CA 91030
United States of America
soren.petersen@ingomar.net

2
1 INTRODUCTION
           
and familiarity with a range of platforms, crowdsourcing is becoming a viable alternative to the
traditional early phases of information gathering in design research. The term crowdsourcing is, in
      -
profit organization, or company proposes to a group of individuals of varying knowledge,
heterogeneity, and number, via a flexible open call, the voluntary undertaking of a task. The
undertaking of the task, of variable complexity and modularity and in which the crowd should
participate bringing their work, money, knowledge and/or experience, always entails mutual benefit.
The user will receive the satisfaction of a given type of need, be it economic, social recognition, self-
esteem, or the development of individual skills, while the crowdsourcer will obtain and utilize to their
advantage that what the user has brought to the venture, whose form will depend on the type of activity
undertaken-Arolas, 2012). Compared to the expensive and time-consuming planning of data
collection in conventional design research, crowdsourced design research offers opportunities for
inexpensive, quick exploration and experimentation with specific homogenous and heterogeneous
groups of networking users. In this article, we propose a Six Step Co-creation Cycle (Petersen,
Santiago, Aitamurto, Spencer and Joo, 2011) method that was applied on thirty-one projects with
varying levels of innovation and complexity. In doing so, we uncovered pros, such as uncovering
biases and collecting new knowledge and cons, such as limited ability to synthesize ideas and the need
to eliminating noise and lack of a network building effect when applying crowdsourcing in the early
phases of design research.
2 CROWDSOURCING
There has been a growing interest in crowdsourcing over the past couple of years, with corporations,
such as IBM and NOKIA developing internal platforms and new providers, such as InnoCentive,
Mechanical Turk and Open IDEO offering crowdsourcing services. Crowdsourcing has successfully
been applied to a wide range of challenges, from clearly defined tasks, such as the design of logos,
development of a engineering solution, writing of a piece of code, gathering ethnographic data and
 
showing that the collective intelligence of a crowd, if its members refrain from communicating with
each other, will converge on a more accurate answer to a challenge than any of its individual expert
members (Surowiecki, 2005). Crowdsourcing concepts for product development further shows that the
crowd can provide more novel and user relevant ideas with only minor payoff in feasibility (Poetz and
Schreier, 2012) and is particularly useful when dealing with long distance searches, where the initiator
has no or little knowledge about the challenge (Afuah and Tucci, 2012). The idea of crowdsourcing is
a particular interesting concept when combined with the Web, where large amount of self-selecting
individuals can be engaged with a minimal effort. In this context crowdsourcing could be a powerful
new application for aiding design research.
3 OVERALL RESEARCH PROCEDURE
        Research  Potentials &
Limitations   examines seven important aspects, five (4.1 to 4.5) based on the study of
E-Arolas and two deemed relevant for research (4.6 and 4.7). The aspects are treated in turn
within the findings.
The research project was conducted over the course of three years, from December 2009 to December
2012, as part of a Stanford Persuasive Technology Lab and Stanford Peace Innovation Lab
investigation into the application of design quantification in the formulation of design briefs for
incremental and breakthrough challenges. Initially, fifteen challenges, phrased as open-ended
questions, focusing on peace innovation and other topics of general public interest were launched to
gain familiarity with, and evaluate the potential of various crowd-sourcing platforms, including
Chaordix, OpenIDEO, Jovoto, Facebook and LinkedIn. In the first exploratory phase, professionals
(designers, journalists, venture capitalists and design researchers) were asked to identify pressing

 

3
the call, post it online and moderate the online discussion until it tapered off, usually after a couple of
weeks. Comments were then collected and insights obtained from the various platforms compared.
Two key lessons were: the importance of (1) a user-friendly interface for reducing the barrier to entry
             
everyone was comfortable with and regularly used Facebook and LinkedIn and the ease of interaction,
resulted in over ten times the number of comments as compared with Chaordix. The pilot project
concluded by selecting the social network platform LinkedIn as the platform for further investigation,
since it combined the capability to reach targeted user groups and familiarity with the platform.
In the second phase, sixteen structured design research projects, applying the SSCC process were
launched, analyzing the performance of the SSCC method.
4 INDIVIDUAL STUDIES AND FINDINGS
The following information describes the seven studies and findings on crowdsourcing performance.
4.1 Who forms the crowd
When crowdsourcing domain-specific topics, understanding participant knowledge and behavior is
essential for generating useful insights. Thus, group engagement and performance was studied as
related to group size.
4.1.1 LinkedIn group size vs. comment rate
Launching thirty-one crowdsourcing challenges on twenty-six LinkedIn groups of various sizes (100
to 200,000 members) and compositions (artists, designers, design educators, design researchers, design
managers, journalist, engineers, entrepreneurs and ergonomists) provided insights into what level of
engagement could be expected. We found that group size and engagement was inversely correlated
            
independent on group size. Furthermore, challenges applying the SSCC method resulted in the same
level of engagement, suggesting that control has little or no influence on engagement. There also
seemed to be an upper limit to engagement, with the engagement dropping off after two to four weeks
and with a maximum engagement level of approximately eight comments per hundred group members.
See Figure 1.
Figure 1. Participation, engagement rate (number of comments) as a function of group
size. Black dots represent crowdsourcing challenges applying the SSCC approach while
gray dots represent challenges with one open-ended question and no summary.

4
4.1.2 Challenge topics in which creative professionals engage
To what type of challenges will creative professionals contribute? Analyzing the engagement for the
SSCC crowdsourcing projects show that creative professionals are primarily motivated by topics
related to design, the top seven most active challenges related to design, the lowest four challenges
relating to business and process. See Figure 2.
Figure 2. Challenge topics (n=16) and the corresponding level of engagement of creative
professionals
4.1.3 Challenges in which LinkedIn groups participate
To what part of design challenges will creative professionals contribute? Analyzing the engagement in
challenges from seven leading LinkedIn design groups on topics of strategic, contextual and execution
show these primarily contribute to challenges related to execution. See Figure 3.
Figure 3. Challenge for design groups (n=7)
divided by type: Strategy, Context and Execution,
with the corresponding level of engagement of
the creative processionals
4.2 What does the crowd have to do?
What is the applicability of crowdsourcing in design research regarding the types of challenge and
feedback, which can be addressed? Analyzing the comments of the initial fifteen pilot projects

5
revealed that participants primarily question the assumptions and uncover biases in the formulation of
the call, bring new knowledge (personal experiences, observations and anecdotal), and references
(links to data and contacts). Participants challenge, support as well as build on previous comments.
However, synthesizing comments into conclusions did not take place, which was then included by the
moderator in the second phase exploration. Following the initial findings and using SSCC, the second
phase, examined incremental and breakthrough challenges as combinations of high/low understanding
of the market and high/low understanding of execution.
4.3 What does the crowd get in return?
What motivates the crowd to sustain and return to participation in future challenges? The initial pilot
project experimented with intrinsic as well as extrinsic rewards and found that extrinsic monetary
rewards in these type of challenges inhibits participation. These findings were confirmed in interviews
with crowdsourcing experts (Waterhouse, 2011) and (Unterberg, 2011). Hence, for the second phase,
intrinsic motivation and continued sharing of findings though group posting of conversation summary
and sharing articles using new social media was used.
4.4 Who is the initiator, what type of calls and what does the initiator get in return?
The initiator was the Design Quantification Lab; a virtual globally distributed multi-functional and
multi-cultural design research group congregating on design research topics of mutual interest.
Challenges posted to the crowd constituted part of parallel projects, applying design quantification on
           
 The quality of knowledge resulting from the challenges proved of a
sufficiently high quality to contribute to papers. Example of insights from crowdsourcing:
 Peace Innovation Journalism: Assuming that the brand value of journalism could be leveraged
in peace innovation we learned that most participants were highly skeptical of the profession
(Petersen, 2012).
 Marketing & Design: Soliciting the crowd for the major hurtles in coordinating marketing and
design showed this to be translating marketing data into actionable criteria.
 Designing for Marginalized Users: Soliciting the crowd for their experiences in designing for
users very different from themselves, highlighted the importance of working with and including
these users in the design process (Petersen and Hussain, 2012).
 Creative Animal Rescue: Soliciting the crowd for insights into how design could assist in animal
rescue revealed that not only does the US have an overpopulation of companion animals due to
profit driven, unregulated breeding practices but also extremely poor marketing of the shelter
animals waiting to be rescued (Petersen, 2012).
 Design and Cultures: Soliciting the crowd on design characteristics of various regions (Europe,
North America, Latin America, Asia, Africa and Scandinavian revealed similarities and
differences between regions and offered cultural explanations for these (Petersen, 2012).
4.5 What type of process and what medium to use?
Based on the experience from the initial pilot projects and auditing of commercial crowdsourcing
platforms the Six-Step Co-creation Cycle was developed in a brainstorming session. The advantages of
the SSCC method, is that it enables the launching of weekly independent small and inexpensive
challenges, making a large number of challenges possible within a short timeframe (2-3 weeks per
challenge). Time investment for setting up, moderating, summarizing, concluding and publishing a
challenge on new social media ranged from one to two days depending on the level of engagement.
The SSCC method consists of the following six steps, see Figure 4.
1) A challenge is formulated by posing a one-line open-ended research question, supported by a
short clarification. For example:
What is American Design?
Every culture has its own beliefs, value and attitudes, reflected in the design of its products, and
services. What adjectives would you assign to American design? Example: Sincerity, Excitement,
Competence, Sophistication, Ruggedness How do these differ from European and Asian design?
The challenge is posted to a number of LinkedIn groups, which contribution and performance is
to be evaluated.

6
Figure 4. Six Step Co-creation Cycle, alternating between expert evaluation and
crowdsourcing ideas on online social networks
2) Discussions on the LinkedIn group are moderated daily and neutral comments are posted for each
contribution to encourage participation. For example:
“Thank you for sharing,” “Good point” or “Could you please elaborate”
Comments are continuously documented in an XLS document and when the discussion taper off,
usually after 2 to 3 weeks, the number of comments is tallied for analysis.
3) Comments are transferred from XLS to a Word document, coded, analyzed and an engaging short
summary story is formulated in approximately 500 words. Popular writing style, combined with
brevity is intended to increase likelihood that challenge participants subsequently read and
commented on the content.
4) Short summary story is posted to the participating LinkedIn groups and the resulting discussion is
monitored as in step 2, usually discussions taper off after a week.
5) Short summary story is updated according to the new comments and the new version is shared
with a minimum of three experts in the area for final evaluation.
6) Finally, - the summary and conclusion is posted to The Huffington Post and the number of
               
document.
4.6 Community building
What is the usefulness of the SSCC method in building a crowdsourcing community? An important
issue for commercial crowdsourcing platforms is building a community of participants for
participation in future challenges. Although leveraging LinkedIn groups ensures an audience, can
momentum be built for engagement on future challenges? To measure the community building effect
of the SSCC method, the number of groups and the number of engagements over the two phases were
examined.
4.6.1 Community building effect over time (group participation)
Longitudinal observations of group activity show no increase in the number groups participating, thus
launching fifteen to thirty-one challenges with regular intervals do not attract more groups. See Figure
5.
4.7 Dissemination
How effective is crowdsourcing for disseminating information?
4.7.1 Dissemination of crowdsourced articles vs. expert articles
Concurrent with publishing the second phase crowdsourcing findings, articles written with individual
experts were published. Analyzing the number of forwarded articles for the two types of articles show
that articles written by experts in general commanded the same number of forwarded articles as the
ones based on findings from crowdsourcing (Mean experts 38 and mean crowdsourced 41, not

7
statistically different). High numbers of forwarded articles is expected to relate to audience interest,
thus, the question of which topics were more likely to be disseminated by crowdsourcing was
examined.
Figure 5. Number of groups participating as a percentage of the total number of groups
the challenges was posted to (groups ranging from 12 to 26). Dark columns represent
breakthrough challenges, while gray columns represent incremental challenges
4.7.2 Dissemination for all articles
Analyzing which topics the crowd disseminate more, shows that articles addressing topics of business
and performance related issues are forwarded more often than other topics. See Figure 7.
Figure 7. Article forwarded most (top 16 of 69), with highest number of forwards, 170
forwards set to index 100. Dark columns represent articles written by experts, while gray
columns represent articles based on crowdsourced challenges

8
4.7.3 Dissemination of topics
Examining what types of articles are of most interest to the creative community in general, we
analyzed the numbers of forwarded articles for categories that were defined by The Huffington Post,

Living and Technology was found to be the three main topics of interest, while Art and Style came in
as numbers four and five. Design related topics are thus unlikely to be forwarded more than other
articles. See Figure 8.
Figure 8. Topics of interests (n=11) for creative professionals as measured by percentage
of forwarded articles
5 SUMMARY AND CONCLUSION
Our study of thirty-one crowdsourced challenges using a social network platform and new social
media shows that applying crowdsourcing in design research, using the Six Step Co-creation Cycle
method, represents a quick, low-cost alternative to the initial phases of design research. The approach
can be used for examining assumptions and uncovering biases, as well as, gathering information. It
provides insights into a wide rage of design topics of varying complexity. These range from fashion
and regional design languages to design for marginalized users and open innovation, contributing to
knowledge from strategy and context to execution.
The advantages of using an established social network platform, such as LinkedIn, for crowdsourcing,
are that no platform development, administration or maintenance or recruitment of participants is
required. Professional communities already exist and can be reached at no expense for platform use.
Since group members are alerted of new challenges and are accustomed to using the platforms, a high
level of engagement is achieved. However, engagement is highly topics-related, such that creative
professionals mainly contribute to design related topics and less to business related topics.
Group size and engagement is invert correlated and large (10,000  100,000 members) and small
groups (500  5,000 members) engage to the same extent in incremental and breakthrough challenges.
Thus, targeting larger and more heterogeneous groups and smaller more homogenous groups does not
present a tradeoff in participation.
Insights from crowdsourced research are disseminated as readily on new social media as that written
by experts on these subjects. Furthermore, observing engagement on LinkedIn (number of comments),
dissemination by The Huffington Post (number of forwarded articles) provides behavioral metrics of
familiarity with topics as well as levels of interest.
Regular crowdsourcing and using SSCC, however, does not facilitate community building, since the
number of groups participating and engagement does not increase over time. This has the advantage

time.
Drawbacks to applying crowdsourcing are that the communities are unable to synthesize
insights/solutions and this has to be done separately by the design research team. Furthermore, the
knowledge gathered varies widely in quality, and verification can be time consuming. An attempt to

9
address this within the SSCC method is to have three or more experts review the summary of the
crowdsourcing challenge and conduct fact checking on questionable findings.
Another is establishing when applying crowdsourcing in new product development is most
advantageous. Our experience with crowdsourcing in research and product development suggests that
the method offers especially valuable insights early in the process. Crowdsourcing has successfully
informed research objectives in research as well as business strategies and business models in product
design.
Further research opportunities would be exploring how to expand the use of crowdsourcing to
entrepreneurial endeavors, where typically the knowledge gap is huge, the risk high, development
speed crucial and resources scarce. One way to investigate this could be to conduct semi-controlled
experiments in startup accelerators and incubators.
ACKNOWLEDGMENTS
The author is indebted to Tina Santiago and Richard Spencer for contributing to the development of
the Six Step Co-creation Cycle method and Dr. Jaewoo Joo and Dr. Martin Steinert, Margarita Quihuis
and Mark Nelson for fruitful cooperation on detailing and testing methods on peace innovation and
design research projects.
REFERENCES
demy of
Management Review, Vol. 37, No. 3, 355375
Convergence: The
International Journal of Research into New Media Technologies, Sage Publications, London, Los
Angeles, New Delhi and Singapore Vol. 14(1): 7590
-Journal of Information
Science, XX (X) pp. 1-14
Petersen S. I. (2011) Insights Using Co-The Huffington Post,
November
Petersen S. I. (2012How Journalists Can Help Design for PeaceThe Huffington Post, January
Petersen S. I. (2012) Creating a Paradigm Shift in Animal Sheltering, The Huffington Post, June
PetEuropean Design -- What Are They Doing Right?The Huffington Post, October
Petersen S. I. and Hussain S. (2012) Design
Management Review, September
Petersen S. I., Santiago T., Aitamurto T., Spencer R. and Joo ing

Professionals in  Journal of Product Innovation Management, 29(2):
245256
Random House, New York
Unterberg B. (2011) Interview, May 17the, Center for Design, Stanford.
Waterhouse N. (2011) Interview, June 3rd, IDEO Palo Alto.