Mark E Whiting

University of Pennsylvania | UP · Department of Computer and Information Science

Just as the telescope ushered in a new era of discovery by broadening astronomers’ sense of sight, digital technology has made it possible to examine human behavior with newfound precision and granularity. Studies of teams now take place in scalable virtual laboratories (Almaatouq et al., 2021); virtual avatars can replace human confederates (de Me...

Commentaries on the target article offer diverse perspectives on integrative experiment design. Our responses engage three themes: (1) Disputes of our characterization of the problem, (2) skepticism toward our proposed solution, and (3) endorsement of the solution, with accompanying discussions of its implementation in existing work and its potenti...

The notion of common sense is invoked so frequently in contexts as diverse as everyday conversation, political debates, and evaluations of artificial intelligence that its meaning might be surmised to be unproblematic. Surprisingly, however, neither the intrinsic properties of common sense knowledge (what makes a claim commonsensical) nor the degre...

Many experimental studies of team performance involve teams working on one specific task or at most a handful of tasks; thus, it is natural to ask how the results of any such study generalize to tasks other than the one(s) studied. Unfortunately, this simple question is generally impossible to answer due to the absence of commensurability among tas...

Understanding the scope, prevalence, and impact of the COVID-19 pandemic response will be a rich ground for research for many years. Key to the response to COVID-19 was the non-pharmaceutical intervention (NPI) measures, such as mask mandates or stay-in-place orders. For future pandemic preparedness, it is critical to understand the impact and scop...

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment...

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a ti...

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment’s specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a ti...

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on s...

Online social actions are often ambiguous, leading us to wonder: Why did this person unfollow me? Why did my friend like this negative content? Such ambiguity is common and perceived as a natural part of our ubiquitous online interactions. However, as online actions are curated and designed by platforms, this ambiguity is, at least in part, somethi...

Deepfakes are synthetic content generated using advanced deep learning and AI technologies. The advancement of technology has created opportunities for anyone to create and share deepfakes much easier. This may lead to societal concerns based on how com�munities engage with it. However, there is limited research available to understand how communit...

Deepfakes are synthetic content generated using advanced deep learning and AI technologies. The advancement of technology has created opportunities for anyone to create and share deepfakes much easier. This may lead to societal concerns based on how communities engage with it. However, there is limited research available to understand how communiti...

As machine learning is used to make strides in med- ical diagnostics, few methods provide heuristics from which human doctors can learn directly. This work introduces a method for leveraging human observable structures, such as macro scale vascular formations, for producing assessments of medical conditions with rela- tively few training cases, and...

Beginning in April 2020, we gathered partial county-level data on non-pharmaceutical interventions (NPIs) implemented in response to the COVID-19 pandemic in the United States, using both volunteer and paid crowdsourcing. In this report, we document the data collection process and summarize our results, to increase the utility of our open data and...

The standard experimental paradigm in the social, behavioral, and economic sciences is extremely limited. Although recent advances in digital technologies and crowdsourcing services allow individual experiments to be deployed and run faster than in traditional physical labs, a majority of experiments still focus on one-off results that do not gener...

Nonverbal actions form an integral part of the interactive experience between users on online platforms. While convenient, these nonverbal online actions, which often take the form of clicking a button to like or share content with others, sacrifice contextual information compared with their in-person and verbal counterparts as they are simplified...

MOOC participants often feel isolated and disconnected from their peers. Navigating meaningful peer interactions, generating a sense of belonging, and achieving social presence are all major challenges for MOOC platforms. MOOC users often rely on external social platforms for such connection and peer interaction, however, off-platform networking of...

MOOC participants often feel isolated and disconnected from their peers. Navigating meaningful peer interactions, generating a sense of belonging, and achieving social presence are all major challenges for MOOC platforms. MOOC users often rely on external social platforms for such connection and peer interaction, however, off-platform networking of...

We report on a systematic review of the landscape of peer assessment in massive open online courses (MOOCs) with papers from 2014 to 2020 in 20 leading education technology publication venues across four databases containing education technology-related papers, addressing three research issues: the evolution of peer assessment in MOOCs during the p...

Virtual labs allow researchers to design high-throughput and macro-level experiments that are not feasible in traditional in-person physical lab settings. Despite the increasing popularity of online research, researchers still face many technical and logistical barriers when designing and deploying virtual lab experiments. While several platforms e...

Understanding team viability --- a team's capacity for sustained and future success --- is essential for building effective teams. In this study, we aggregate features drawn from the organizational behavior literature to train a viability classification model over a dataset of 669 10-minute text conversations of online teams. We train classifiers t...

Was a problematic team always doomed to frustration, or could it have ended another way? In this paper, we study the consistency of team fracture: a loss of team viability so severe that the team no longer wants to work together. Understanding whether team fracture is driven by the membership of the team, or by how their collaboration unfolded, mot...

Understanding team viability -- a team's capacity for sustained and future success -- is essential for building effective teams. In this study, we aggregate features drawn from the organizational behavior literature to train a viability classification model over a dataset of 669 10-minute text conversations of online teams. We train classifiers to...

Virtual labs allow researchers to design high-throughput and macro-level experiments that are not feasible in traditional in-person physical lab settings. Despite the increasing popularity of online research, researchers still face many technical and logistical barriers when designing and deploying virtual lab experiments. While several platforms e...

A team's early interactions are influential: small behaviors cascade, driving the team either toward successful collaboration or toward fracture. Would a team be more viable if it could undo initial interactional missteps and try again? We introduce a technique that supports online and remote teams in creating multiple parallel worlds: the same tea...

Was a problematic team always doomed to frustration, or could it have ended another way? In this paper, we study the consistency of team fracture: a loss of team viability so severe that the team no longer wants to work together. Understanding whether team fracture is driven by the membership of the team, or by how their collaboration unfolded, mot...

Accurate task pricing in microtask marketplaces requires substantial effort via trial and error, contributing to a pattern of worker underpayment. In response, we introduce Fair Work, enabling requesters to automatically pay their workers minimum wage by adding a one-line script tag to their task HTML on Amazon Mechanical Turk. Fair Work automatica...

Paid crowdsourcing platforms suffer from low-quality work and unfair rejections, but paradoxically, most workers and requesters have high reputation scores. These inflated scores, which make high-quality work and workers difficult to find, stem from social pressure to avoid giving negative feedback. We introduce Boomerang, a reputation system for c...

Massive Open Online Courses (MOOCs) often use peer grading which enables the use of open ended questions in assignments at scale. Open ended questions enhance students' creativity and improve cognition. However, this peer grading includes drawback such as missing or low quality feedback, and inconsistency between peer graders. In design studios and...

Massive Open Online Courses (MOOCs) often use peer grading which enables the use of open ended questions in assignments at scale. Open ended questions enhance students’ creativity and improve cognition. However, this peer grading includes drawback such as missing or low quality feedback, and inconsistency between peer graders. In design studios and...

Rule based analysis of biological phenomena has rich potential but has been limited by our lack of methods for establishing systems of rules that discretely represent arbitrary structured data. Graph grammar is a formalism affording this kind of rule based representation of arbitrary information, and many tools have been developed to use graph gram...

The use of grammars in design and analysis has been set back by the lack of automated ways to induce them from arbitrarily structured datasets. Machine translation methods provide a construct for inducing grammars from coded data which have been extended to be used for design through pre-coded design data. This work introduces a four-step process f...

Low-quality results have been a long-standing problem on microtask crowdsourcing platforms, driving away requesters and justifying low wages for workers. To date, workers have been blamed for low-quality results: they are said to make as little effort as possible, do not pay attention to detail, and lack expertise. In this paper, we hypothesize tha...

Massive Open Online Courses (MOOCs) use peer assessment to grade open ended questions at scale, allowing students to provide feedback. Relative to teacher based grading, peer assessment on MOOCs traditionally delivers lower quality feedback and fewer learner interactions. We present the identified peer review (IPR) framework, which provides non-bli...

The success of crowdsourcing markets is dependent on a strong foundation of trust between workers and requesters. In current marketplaces, workers and requesters are often unable to trust each other's quality, and their mental models of tasks are misaligned due to ambiguous instructions or confusing edge cases. This breakdown of trust typically ari...

Grammars are useful for representing systems of design patterns, however formulating good grammars is not straightforward in many contexts due to challenges of representation and scope. This challenge has been identified as one of the 3 goals for computerized use of shape grammars: grammar inference. This work introduces a highly flexible mechanism...

Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized wor...

Paid crowdsourcing platforms suffer from low-quality work and unfair rejections, but paradoxically, most workers and requesters have high reputation scores. These inflated scores, which make high-quality work and workers difficult to find, stem from social pressure to avoid giving negative feedback. We introduce Boomerang, a reputation system for c...

This paper develops a formal approach to investigate the evolution of a Korean traditional pattern, Bosangwhamun. The approach employs the structure of symbolic memes embedded in the pattern as a framework of hierarchical decomposition of a pattern to describe an evolutionary development process of a given pattern with a set of rules in shape gramm...

Cloud Computing, a phrase originally coined to represent telephony grids, is now used to generically describe service systems based on distributed computing of some description. In recent years Cloud Computing has gained substantial coverage, partially due to a flood of web-based services and to the move from hardware innovation to network and soft...