Steven P. Dow

University of California, San Diego | UCSD · Department of Cognitive Science

We believe that our education systems are currently not capable of evolving at the rate necessary to meet the challenges presented by rapidly changing technology epitomized by Industry 4.0 and the digital transformation. Our research project goal is to pilot a learning laboratory that can be used to conduct the necessary experiments to create and a...

Making sense of large unstructured problem spaces is cognitively demanding. Structure can help, but adding structure to a problem space also takes significant effort. ProbMap is a novel application for automatically constructing a design gallery from unstructured text input. Given a list of problem statements, ProbMap extracts and semantically grou...

Civic problems are often too complex to solve through traditional top-down strategies. Various governments and civic initiatives have explored more community-driven strategies where citizens get involved with defining problems and innovating solutions. While certain people may feel more empowered, the public at large often does not have accessible,...

Due to challenges around low-quality comments and misinformation, many news outlets have opted to turn off commenting features on their websites. The New York Times (NYT), on the other hand, has continued to scale up its online discussion resources to reach large audiences. Through interviews with the NYT moderation team, we present examples of how...

Creating truly original ideas requires extensive knowledge of existing ideas. Navigating prior examples can help people to understand what has already been done and to assess the quality of their own ideas through comparison. The creativity literature has suggested that the conceptual distance between a proposed solution and a potential inspiration...

Many organizations have adopted design processes that integrate community voices to discover the real problems that communities face. Online discussion forums offer a familiar and flexible technology that can help facilitate discussion around problems and potential solutions. However, we lack understanding about what information community members s...

Team dating, or small-group interactions, can expose people to diverse perspectives and inform the potential for longer-term collaboration. However, rapidly configuring groups and facilitating interactions among strangers can be difficult, especially in co-located settings. We present ProtoTeams, a system that leverages personal mobile devices to s...

Creative workers frequently turn to online critique communities for feedback on their work. While past research has focused primarily on how to yield better feedback from providers, less is known about the strategies feedback seekers use to engage providers and request feedback. We present two studies to explore the feedback exchange dynamics betwe...

News websites can facilitate global discussions about civic issues, but the financial cost and burden of moderating these forums has forced many to disable their commenting systems. In this paper, we consider the role that data visualizations play in online discussion around a civic issue, through an analysis of how people talk about climate change...

Group facilitators often organize large co-located crowds into small groups where people can more effectively interact and learn from each other. However, current approaches for facilitating small group interactions typically do not capture data about the rapport and future potential of interacting groups. We introduce ProtoTeams, a system designed...

As the DIS community increasingly seeks to address social impact issues, it becomes important to examine the assumptions behind our methods to increase the likelihood of positive effects and reduce negative unintended consequences. The purpose of this workshop is to engage the design community in exploring, defining, and, if deemed valuable, advanc...

Advancements in digital civics have enabled leaders to engage and gather input from a broader spectrum of the public. However, less is known about the analysis process around community input and the challenges faced by civic leaders as engagement practices scale up. To understand these challenges, we conducted 21 interviews with leaders on civic-or...

Exposing people to concepts created by others can inspire novel combinations of concepts, or conversely, lead people to simply emulate others. But how does the type of exposure affect creative outcomes in online collaboration where dyads interact for short tasks? In this paper, we study the creative outcomes of dyads working together online on a sl...

Peer feedback is essential for learning in project-based disciplines. However, students often need guidance when acting as either a feedback provider or a feedback receiver, both to gain from peer feedback and to criticize their peers' work. This paper explores how to more effectively scaffold this exchange such that peers more deeply engage in the...

From Fortune 500 companies to local communities, organizations often strive to build a shared understanding about complex problems. Design competitions provide a compelling approach to create incentives and infrastructure for gathering insights about a problem-space. In this paper, we present an analysis of a two-month civic design competition focu...

This workshop invites the CSCW community to explore hybrid events - large collocated events where technology is used to support audience participation. We argue that the technology landscape has changed since the early studies in CSCW towards this context. Therefore, the research foci must similarly change and focus on studying the practices or pro...

In this paper, we present the findings of a pilot study where we randomly resurfaced professional designers' own archived ideas to them over a period of three weeks. We find that resurfacing ideas can provide reflective and creative value to designers by encouraging them to reflect on their old ideas, reflect on themselves as practitioners, and as...

This paper presents the results of an extensive qualitative study investigating how professional designers utilize personal idea archives. While we know that designers archive creative ideas in different formats and on different platforms, we know little about if and how designers utilize these idea archives in their daily practice. Through a serie...

Examples provide a source of inspiration for creating designs, but can they help improve the feedback process? Supplementing design feedback with examples could help recipients see issues clearly, identify concrete steps for improvement, and integrate novel ideas. Two online studies investigated how to support novices providing feedback on visual p...

While urban design affects the public, most people do not have the time or expertise to participate in the process. Many online tools solicit public input, yet typically limit interaction to collecting complaints or early-stage ideas. This paper explores how to engage the public in more complex stages of urban design without requiring a significant...

Groups often face difficulty reaching consensus. For complex decisions with multiple criteria, verbal and written discourse alone may impede groups from pinpointing and moving past fundamental disagreements. To help support consensus building, we introduce ConsensUs, a novel visualization tool that highlights disagreement by asking group members to...

Increasing classroom sizes and decreasing financial and human resources have encouraged educators to seek innovative strategies to manage large classrooms. Several instructors have begun using web-based peer reviews as a way to increase open-ended feedback. Recent work in design-based classes has revealed that students struggle to provide meaningfu...

Design, a cornerstone of engineering education, necessarily involves a practical training approach, which often requires educators to navigate non-traditional learning environments. Ambiguity in design is the result of the existence of multiple solutions to a given problem, and the need to find an optimal solution most often based on incomplete inf...

Design ideas often come from sources of inspiration (e.g., analogous designs, prior experiences). In this paper, we test the popular but unevenly supported hypothesis that conceptually distant sources of inspiration provide the best insights for creative production. Through text analysis of hundreds of design concepts across a dozen different desig...

Team-based learning is a structured, small-group learning method that has been associated with many positive outcomes in traditional classroom settings. However, relatively little research has focused on how to form and support teams within online learning platforms, such as Massive Open Online Courses (MOOCs). A number of challenges arise for team...

Classroom role-play is an interactive learning technique with a long history of success, but current attempts to augment it with technology limit the very interactions that make this technique successful. For example, digital role-play games often engage individual students at a computer, rather than creating rich social interactions among students...

Collaborative ideation systems can help people generate more creative ideas by exposing them to ideas different from their own. However, there are competing theoretical views on whether and when such exposure is helpful. Associationist theory suggests that exposing ideators to ideas that are semantically far from their own maximizes novel combinati...

Online crowds can help infuse creativity into the design process, but traditional strategies for leveraging them, such as large-scale ideation platforms, require time and organizational effort in order to obtain results. We propose a new method for crowd-based ideation that simplifies the process by having smaller crowds join in-person ideators dur...

Feedback from diverse audiences can contain ambiguity and contradictions, making it difficult to interpret and act on. To promote deeper interpretation of feedback, we tested the effects of combining a reflection activity and reviewing external feedback for an iterative design task. Designers (N=90) created a design and revised it after a) performi...

Designers are increasingly leveraging online crowds; yet, on-line contributors may lack the expertise, context, and sensitivity to provide effective critique. Rubrics help feedback providers but require domain experts to write them and may not generalize across design domains. This paper introduces and tests a novel semi-automated method to support...

Peer feedback is a central activity for project-based design education. The prevalence of devices carried by students and the emergence of novel peer feedback systems enables the possibility of collecting and sharing feedback immediately between students during class. However, pen and paper is thought to be more familiar, less distracting for stude...

As demand for design education increases, instructors are struggling to provide timely, personalized feedback for student projects. Gathering feedback from classroom peers and external crowds offer scalable approaches, but there is little evidence of how they compare. We report on a study in which students (n=127) created early- and late-stage prot...

To meet the demand for authentic, timely, and affordable feedback, researchers have explored technologies to connect designers with feedback providers online. While researchers have implemented mechanisms to improve the content of feedback, most systems for online feedback exchange do not support an end-to-end cycle, from help-seeking to sense-maki...

Workers' level of skill on online labor platforms varies greatly. To manage quality, requesters often decompose complex jobs into simple, repetitive, low-pay micro-tasks. Re-questers get the jobs done, but workers have little opportunity to learn and prepare for more demanding and better-pay jobs. This paper explores how we can apply learning scien...

Feedback is information that can improve task performance. Online communities, educational forums, and crowd-based feedback platforms all support feedback exchange among a more diverse set of sources than ever before, with greater control over how to moderate this exchange. In this work, we study how the power relationship between the source and re...

Forming work teams involves matching people with complementary skills and personalities, but requires obtaining such data a priori. We introduce team dating, where people interact on brief tasks before working with a dedicated partner for longer, more complex tasks. We studied team dating through two online experiments. In Experiment 1, workers fro...

Groups often face difficulty reaching consensus. For complex decisions with multiple latent criteria, discourse alone may impede groups from pinpointing fundamental disagreements. To help support a consensus building process, we introduce ConsensUs, a novel visualization tool that highlights disagreements in comparative decisions. The tool fa cilit...

Prior work on creativity support tools demonstrates how a computational semantic model of a solution space can enable interventions that substantially improve the number, quality and diversity of ideas. However, automated semantic modeling often falls short when people contribute short text snippets or sketches. Innovation platforms can employ huma...

Increasingly, designers seek feedback on their designs from crowd platforms such as social networks, Web forums, and paid task markets which demand different amounts of social capital, financial resources, and time. Yet it is unknown how the choice of crowd platform affects feedback generation. We conducted an online study where designers created i...

Peer feedback systems enable students to get feedback without substantially burdening the instructor. However, current systems typically ask students to provide feedback after class; this introduces challenges for ensuring relevant, timely, diverse, and sufficient amounts of feedback, and reduces time available for student reflection. This paper ex...

To create a satisfying social learning experience, an emerging challenge in educational data mining is to automatically assign students into effective learning teams. In this paper, we utilize discourse data mining as the foundation for an online team-formation procedure. The procedure features a deliberation process prior to team assignment, where...

The physical constraints of smartwatches limit the range and complexity of tasks that can be completed. Despite interface improvements on smartwatches, the promise of enabling productive work remains largely unrealized. This paper presents WearWrite, a system that enables users to write documents from their smartwatches by leveraging a crowd to hel...

Large-scale idea generation platforms often expose ideators to previous ideas. However, research suggests people generate better ideas if they see abstracted solution paths (e.g., descriptions of solution approaches generated through human sensemaking) rather than being inundated with all prior ideas. Automated and semi-automated methods can also o...

Online crowds have the potential to do more complex work in teams, rather than as individuals. However, at such a large scale, team formation can be difficult to coordinate. (How) can we rely on the crowd itself to organize into effective teams? Our research explores a strategy for "team dating", a self-organized crowd team formation approach where...

When personalities clash, teams operate less effectively. Personality differences affect face-to-face collaboration and may lower trust in virtual teams. For relatively short-lived assignments, like those of online crowdsourcing, personality match- ing could provide a simple, scalable strategy for effective team formation. However, it is not clear...

Online crowds are a promising source of new innovations. However, crowd innovation quality does not always match its quantity. One way to improve quality is to enable experts to provide personalized feedback. However, this scales poorly, and may lead to premature convergence during creative work. To deal with these issues, we present IdeaGens, a cr...

Social science researchers spend significant time annotating behavioral events in video data in order to quantitatively assess interactions [2]. These behavioral events may be instantaneous changes, continuous actions that span unbounded periods of time, or behaviors that would be best described by severity or other scalar ratings. The complexity...

One main challenge in large creative online communities is helping their members find inspirational ideas from a large pool of ideas. A high-level approach to address this challenge is to create a synthesis of emerging solution space that can be used to provide participants with creative and diverse inspirational ideas of others. Existing approache...

Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted an experiment with a 2x2 factorial design. Stud...

Online crowds are a promising source of new innovations. However, crowd innovation quality does not always match its quantity. In this paper, we explore how to improve crowd innovation with real-time expert guidance. One approach would for experts to provide personalized feed-back, but this scales poorly, and may lead to premature convergence durin...

In this paper we introduce a paradigm for completing complex tasks from wearable devices by leveraging crowdsourcing, and demonstrate its validity for academic writing. We explore this paradigm using a collaborative authoring system, called WearWrite, which is designed to enable authors and crowd workers to work together using an Android smartwatch...

Emerging online ideation platforms with thousands of example ideas provide an important resource for creative production. But how can ideators best use these examples to create new innovations? Recent work has suggested that not just the choice of examples, but also the timing of their delivery can impact creative outcomes. Building on existing cog...

In user-centered design processes, one of the most important tasks is to synthesize information from user research into insights and a shared point of view among team members. This paper explores the synthesis process and opportunities for providing computational support. First, we present interviews on the common practices and challenges of inform...

Coding behavioral video is an important method used by researchers to understand social phenomenon. Unfortunately, traditional hand-coding approaches can take days or weeks of time to complete. Recent work has shown that these tasks can be completed quickly by leveraging the parallelism of large online crowds, but using the crowd introduces new con...

Online collaboration tools enable developers of interactive systems to quickly reach potential users for usability testing. Can these technologies serve designers who seek feedback on user needs during the earliest stages of design? Online needfinding may help designers create products and services that can target a more diverse user population. To...

A growing number of large collaborative idea generation platforms promise that by generating ideas together, people can create better ideas than any would have alone. But how might these platforms best leverage the number and diversity of contributors to help each contributor generate even better ideas? Prior research suggests that seeing particula...

Crowd feedback systems offer designers an emerging approach for improving their designs, but there is little empirical evidence of the benefit of these systems. This paper reports the results of a study of using a crowd feedback system to iterate on visual designs. Users in an introductory visual design course created initial designs satisfying a d...

Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and...

Behavioral researchers spend considerable amount of time coding video data to systematically extract meaning from subtle human actions and emotions. In this paper, we present Glance, a tool that allows researchers to rapidly query, sample, and analyze large video datasets for behavioral events that are hard to detect automatically. Glance takes adv...

Constructing a good conference schedule for a large multi-track conference needs to take into account the preferences and constraints of organizers, authors, and attendees. Creating a schedule which has fewer conflicts for authors and attendees, and thematically coherent sessions is a challenging task. Cobi introduced an alternative approach to con...

Many crowd ideation systems seek to gather scores of ideas from people online. However, this often leads to many bad ideas and duplication. A dedicated facilitator who guides exploration of the solution space is a common and effective strategy for optimizing ideation in face-to-face brainstorming, but has not yet been explored in computer-supported...

Behavioral researchers code video to extract systematic meaning from subtle human actions and emotions. While this has traditionally been done by analysts within a research group, recent methods have leveraged online crowds to massively parallelize this task and reduce the time required from days to seconds. However, using the crowd to code video i...

By definition, human-centered design relies on interaction with users. While interacting with users within industry can be challenging, fostering these interactions in a classroom setting can be even more difficult. This qualitative study explores the use of crowd-based design activities as a way to support student-user interactions online. We moti...

User-centered designers often seek to synthesize data from user research into insights and a shared point of view among team members. This paper explores the synthesis process and opportunities for providing computational support. First, we present interviews with novice and expert designers on the common practices and challenges of syn-thesis. Bas...

Organizing conference sessions around themes improves the experience for attendees. However, the session creation process can be difficult and time-consuming due to the amount of expertise and effort required to consider alternative paper groupings. We present a collaborative web application called Frenzy to draw on the efforts and knowledge of an...

Behavioral coding is a common technique in the social sciences and human computer interaction for extracting meaning from video data [3]. Since computer vision cannot yet reliably interpret human actions and emotions, video coding remains a time-consuming manual process done by a small team of researchers. We present Glance, a tool that allows rese...

By recruiting large numbers of people online to perform small tasks, researchers can perform important assessments that are hard to obtain otherwise, at a very reasonable cost and speed. These tasks include assessing quality, reading characters that OCR readers can’t decipher, labeling photographs, and even answering questions in a survey. Care mus...

In crowd-collaborative innovation platforms, other contributors' ideas can serve as sources of inspiration for creative ideas, but what patterns of interactions with others' ideas are most helpful? We investigate the hypothesis that building on inspiration sources that are conceptually far from one's target domain are most helpful, a popular hypoth...

Analysts synthesize complex, qualitative data to uncover themes and concepts, but the process is time-consuming, cognitively taxing, and automated techniques show mixed success. Crowdsourcing could help this process through on-demand harnessing of flexible and powerful human cognition, but incurs other challenges including limited attention and exp...

In modern crowdsourcing markets, requesters face the challenge of training and managing large transient workforces. Requesters can hire peer workers to review others' work, but the value may be marginal, especially if the reviewers lack requisite knowledge. Our research explores if and how workers learn and improve their performance in a task domai...

People who create visual designs often struggle to find high-quality critique outside a firm or classroom, and current online feedback solutions are limited. We created a system called CrowdCrit which leverages paid crowdsourcing to generate and visualize high-quality visual design critique. Our work extends prior crowd feedback research by focusin...

This article presents the results of an online creativity experi-ment (N = 81) that examines the effect of example timing on creative output. In the between-subjects experiment, partici-pants drew animals to inhabit an alien Earth-like planet while being exposed to examples early, late, or repeatedly during the experiment. We find that exposure to...

Current assessment instruments for socio-emotional skills typically focus on diagnosing dysfunction and need to be administered by teachers and parents. To explore socio-emotional learning in novel environments, the learning sciences community needs scalable, easy-to-administer research instruments that measure growth in normal children. In this pa...