Karrie Karahalios

Singapore-MIT Alliance · MIT Media Lab

Research into community content moderation often assumes that moderation teams govern with a single, unified voice. However, recent work has found that moderators disagree with one another at modest, but concerning rates. The problem is not the root disagreements themselves. Subjectivity in moderation is unavoidable, and there are clear benefits to...

Social media sites like Reddit, Discord, and Clubhouse utilize a community-reliant approach to content moderation. Under this model, volunteer moderators are tasked with setting and enforcing content rules within the platforms' sub-communities. However, few mechanisms exist to ensure that the rules set by moderators reflect the values of their comm...

Instructors using algorithmic team formation tools must decide which criteria (e.g., skills, demographics, etc.) to use to group students into teams based on their teamwork goals, and have many possible sources from which to draw these configurations (e.g., the literature, other faculty, their students, etc.). However, tools offer considerable flex...

A conversational agent (CA) effectively facilitates online group discussions at scale. However, users may have expectations about how well the CA would perform that do not match with the actual performance, compromising technology acceptance. We built a facilitator CA that detects a member who has low contribution during a synchronous group chat di...

Legal crowdfunding is an emerging domain where lawyers and individuals raise funds to fight legal actions. To study how prospective donors can verify the credibility of legal campaigns, we analyzed the conversations surrounding these campaigns on Facebook. We discovered three primary themes associated with the perceptions of the contributors of leg...

Informed consent is a core cornerstone of ethics in human subject research. Through the informed consent process, participants learn about the study procedure, benefits, risks, and more to make an informed decision. However, recent studies showed that current practices might lead to uninformed decisions and expose participants to unknown risks, esp...

Headlines play a critical role in how users perceive articles. But many headline publishers craft headlines in ways that either attract clicks in an attempt to earn ad revenue, or misinform users or manipulate their opinions for malicious intents. Such headlines can do harm since many users simply skim and share headlines without reading the articl...

A sense of community is important in encouraging people to contribute to a variety of causes and the communities that support them. Researchers have identified website design features that can engender a sense of community on sites to promote contributions. However, most findings about design features are based on observational empirical research t...

Data visualization is the primary means by which data analysts explore patterns, trends, and insights in their data. Unfortunately, existing visual analytics tools offer limited expressiveness and scalability when it comes to searching for visualizations over large datasets, making visual data exploration labor-intensive and time-consuming. In this...

Conversational surveys, where an agent asks open-ended questions through natural language interfaces, offer a new way to collect information from people. A good follow-up question in a conversational survey prompts high-quality information and delivers engaging experiences. However, generating high-quality follow-up questions on the fly is a non-tr...

How do power dynamics and value conflicts affect our ability to design and maintain socio-technical algorithmic processes? In this paper, we study the SIGCHI student volunteer (SV) selection process that uses a weighted semi-randomized algorithm to recruit a desired pool of volunteers. Our interviews with the community members showed that the proce...

Content moderation systems for social media have had numerous issues of bias, in terms of race, gender, and ability among many others. One proposal for addressing such issues in automated decision making is by designing for contestability, whereby users can shape and influence how decisions are made. In this study, we conduct a series of participat...

Data visualization is the primary means by which data analysts explore patterns, trends, and insights in their data. Unfortunately, existing visual analytics tools offer limited expressiveness and scalability when it comes to searching for visualizations over large datasets, making visual data exploration labor-intensive and timeconsuming. We first...

Creative live streams, where artists or designers demonstrate their creative process, have emerged as a unique and popular genre of live streams due to the real-time interactivity they afford. However, streamer-viewer interactions on most live streaming platforms only enable users to utilize text and emojis to communicate, which limits what viewers...

Surveys are a common instrument to gauge self-reported opinions from the crowd for scholars in the CSCW community, the social sciences, and many other research areas. Researchers often use surveys to prioritize a subset of given options when there are resource constraints. Over the past century, researchers have developed a wide range of surveying...

Visualization recommendation (VisRec) systems provide users with suggestions for potentially interesting and useful next steps during exploratory data analysis. These recommendations are typically organized into categories based on their analytical actions, i.e., operations employed to transition from the current exploration state to a recommended...

Spreadsheet systems are by far the most popular platform for data exploration on the planet, supporting millions of rows of data. However, exploring spreadsheets that are this large via operations such as scrolling or issuing formulae can be overwhelming and error-prone. Users easily lose context and suffer from cognitive and mechanical burdens whi...

As colleges and universities continue their commitment to increasing access to higher education through offering education online and at scale, attention on teaching open-ended subjects online and at scale, mainly the arts, humanities, and the social sciences, remains limited. While existing work in scaling open-ended courses primarily focuses on t...

Algorithms are ubiquitous and critical sources of information online and increasingly act as gatekeepers for users accessing or sharing information about virtually any topic. This includes information about their personal lives and those of friends and family, news and politics, entertainment, and even health and well-being. As a result, algorithmi...

While researchers have developed rigorous practices for offline housing audits to enforce the US Fair Housing Act, the online world lacks similar practices. In this work we lay out principles for developing and performing online fairness audits. We demonstrate a controlled sock-puppet audit technique for building online profiles associated with a s...

As algorithmic (and particularly machine learning) decision making systems become both more widespread and make more important decisions, there are growing concerns about their embedded values and ability to establish legitimacy among decision subjects. We argue that designing for contestability in these systems can assist in surfacing values, alig...

Videos are essential for successful crowdfunding campaigns. However, without knowledge of the underlying persuasion factors, novice entrepreneurs may find it difficult to optimize their videos for success. This paper presents VidLyz, a novel assistive tool that allows users to explore the implications of audience-engagement persuasion factors in th...

With computational algorithms making an increasing number of deeply consequential, and often problematic judgments on our behalf, there is a growing interest in slowing down technology to encourage users to reflect on judgments made by algorithms. Prior work in slow technology has established slowness as an agent of reflection and serendipity; howe...

In content-based online platforms, use of aggregate user feedback (say, the sum of votes) is commonplace as the "gold standard" for measuring content quality. Use of vote aggregates, however, is at odds with the existing empirical literature, which suggests that voters are susceptible to different biases-reputation (e.g., of the poster), social inf...

This paper examines the use of the abstract comic form for persuading online charitable donations. Persuading individuals to contribute to charitable causes online is hard and responses to the appeals are typically low; charitable donations share the structure of public goods dilemmas where the rewards are distant and non-exclusive. In this paper,...

Spatial knowledge about the environment often helps people accomplish their navigation and wayfinding tasks more efficiently. Off-the-shelf mobile navigation applications often focus on guiding people to go between two locations, ignoring the importance of learning spatial knowledge. Drawing on theories and findings from the area of learning spatia...

Visual query systems (VQSs) empower users to interactively search for line charts with desired visual patterns, typically specified using intuitive sketch-based interfaces. Despite decades of past work on VQSs, these efforts have not translated to adoption in practice, possibly because VQSs are largely evaluated in unrealistic lab-based settings. T...

Visualizations are emerging as a means of spreading digital misinformation. Prior work has shown that visualization interpretation can be manipulated through slanted titles that favor only one side of the visual story, yet people still think the visualization is impartial. In this work, we study whether such effects continue to exist when titles an...

It is often assumed that visual cues, which highlight specific parts of a visualization to guide the audience's attention, facilitate visualization storytelling and presentation. This assumption has not been systematically studied. We present an in-lab experiment and a Mechanical Turk study to examine the effects of integral and separable visual cu...

Algorithms exert great power in curating online information, yet are often opaque in their operation, and even existence. Since opaque algorithms sometimes make biased or deceptive decisions, many have called for increased transparency. However, little is known about how users perceive and interact with potentially biased and deceptive opaque algor...

Users frequently use search systems on the Web as well as online social media to learn about ongoing events and public opinion on personalities. Prior studies have shown that the top-ranked results returned by these search engines can shape user opinion about the topic (e.g., event or person) being searched. In case of polarizing topics like politi...

Identifying trendline visualizations with desired patterns is a common and fundamental data exploration task. Existing visual analytics tools offer limited flexibility and expressiveness for such tasks, especially when the pattern of interest is under-specified and approximate, and do not scale well when the pattern searching needs are ad-hoc, as i...

How can instructors group students into teams that interact and learn effectively together? One strand of research advocates for grouping students into teams with "good" compositions such as skill diversity. Another strand argues for deploying team-building activities to foster interpersonal relations like psychological safety. Our work synthesizes...

Social media sites use different labels to help users find and select news feeds. For example, Blue Feed, Red Feed, a news feed created by the Wall Street Journal, use stance labels to separate news articles with opposing political ideologies to help people explore diverse opinions. To combat the spread of fake news, Facebook has experimented with...

Spatial user interfaces that help people navigate often focus on turn-by-turn instructions, ignoring how they may help incidental learning of spatial knowledge. Drawing on theories and findings from the area of spatial cognition, the current paper aims to understand how turn-by-turn instructions and relative location updates can help incidental lea...

Finding visualizations with desired patterns is a common goal during data exploration. However, due to the limited expressiveness and flexibility of existing visual analytics systems, pattern-based querying of visualizations has largely been a manual process. We demonstrate ShapeSearch, a system that enables users to express their desired patterns...

People desire to present themselves favorably to others. However, medical crowdfunding beneficiaries are often expected to present their dire medical conditions and financial straits to solicit financial support. To investigate how beneficiaries convey their situation on medical crowdfunding pages and how contributors perceive the presented informa...

Advertisers develop algorithms to select the most relevant advertisements for users. However, the opacity of these algorithms, along with their potential for violating user privacy, has decreased user trust and preference in behavioral advertising. To mitigate this, advertisers have started to communicate algorithmic processes in behavioral adverti...

Slanted framing in news article titles induce bias and influence recall. While recent studies found that viewers focus extensively on titles when reading visualizations, the impact of titles in visualization remains underexplored. We study frames in visualization titles, and how the slanted framing of titles and the viewer's pre-existing attitude i...

Algorithmic prioritization is a growing focus for social media users. Control settings are one way for users to adjust the prioritization of their news feeds, but they prioritize feed content in a way that can be difficult to judge objectively. In this work, we study how users engage with difficult-to-validate controls. Via two paired studies using...

Donation-based crowdfunding platforms have an increasing number of campaigns on socially stigmatized topics. These platforms' widespread online reachability and the large flow of monetary donations have the potential to shape individuals' opinions by influencing their perceptions. However, little research has been done to investigate whether these...

In traditional usability studies, researchers talk to users of tools to understand their needs and challenges. Insights gained via such interviews offer context, detail, and background. Due to costs in time and money, we are beginning to see a new form of tool interrogation that prioritizes scale, cost, and breadth by utilizing existing data from o...

In studying the increasing role that opaque, algorithmically-driven systems, such as social media feeds, play in society and people's everyday lives, user folk theories are emerging as one powerful lens with which to examine the relationship between user and algorithmic system. Folk theories allow researchers to better see from users' own perspecti...

In traditional usability studies, researchers talk to users of tools to understand their needs and challenges. Insights gained via such interviews offer context, detail, and background. Due to costs in time and money, we are beginning to see a new form of tool interrogation that prioritizes scale, cost, and breadth by utilizing existing data from o...

This paper presents a comparative study of two webtools developed to capture engagement via visualization of coordinated communication behavior in children with autism. A clear preference arose for different tasks based on behavior granularity emphasis in the two visualizations. The survey and interview results further revealed the importance of sh...

The increasing availability of rich and complex data in a variety of scientific domains poses a pressing need for tools to enable scientists to rapidly make sense of and gather insights from data. One proposed solution is to design visual query systems (VQSs) that allow scientists to search for desired patterns in their datasets. While many existin...

The best actors, particularly classic Shakespearian actors, are experts at vocal expression. With prosodic inflection, change of voice quality, and non-textual utterances, they communicate emotion, emphasize ideas, create drama, and form a complementary language which works with the text to tell the story in the script. To begin to study selected e...

Data visualization is an effective mechanism for identifying trends, insights, and anomalies in data. On large datasets, however, generating visualizations can take a long time, delaying the extraction of insights, hampering decision making, and reducing exploration time. One solution is to use online sampling-based schemes to generate visualizatio...

Presenters, such as analysts briefing to an executive committee, often use visualizations to convey information. In these cases, providing clear visual guidance is important to communicate key concepts without confusion. This paper explores visual cues that guide attention to a particular area of a visualization. We developed a visual cue taxonomy...

Awareness of bias in algorithms is growing among scholars and users of algorithmic systems. But what can we observe about how users discover and behave around such biases? We used a cross-platform audit technique that analyzed online ratings of 803 hotels across three hotel rating platforms and found that one site’s algorithmic rating system biased...

Understanding the factors that persuade backers to donate to research projects has become increasingly important with the rising popularity of scientific crowdfunding. Although there are many similarities between enterprise and scientific crowdfunding, some factors differentiate these two forms of crowdfunding. One such factor is the use of endorse...

Instructors are increasingly using algorithmic tools for team formation, yet little is known about how these tools are applied or how students and instructors perceive their use. We studied a representative team formation tool (CATME) in eight project-based courses. An instructor uses the tool to form teams by surveying students' working styles, sk...

Awareness of bias in algorithms is growing among scholars and users of algorithmic systems. But what can we observe about how users discover and behave around such biases? We used a cross-platform audit technique that analyzed on-line ratings of 803 hotels across three hotel rating platforms and found that one site's algorithmic rating system biase...

Search systems in online social media sites are frequently used to find information about ongoing events and people. For topics with multiple competing perspectives, such as political events or political candidates, bias in the top ranked results significantly shapes public opinion. However, bias does not emerge from an algorithm alone. It is impor...

Medical crowdfunding helps patients receive financial support from their distributed social networks online. However, little is known about who the patient's supporters are, what support they provide, and why. To address this, we interviewed fifteen people involved in medical crowdfunding, including both beneficiaries and supporters. We found that...

To successfully raise money using crowdfunding, it is important for a campaign to communicate ideas or products effectively to the potential backers. One of the lesser explored but powerful components of a crowdfunding campaign is the campaign video. To better understand how videos affect campaign outcomes, we analyzed videos from 210 Kickstarter c...

Music therapy (MT) is a therapeutic practice where a therapist uses music to enhance the life quality for their patients. Children have an innate enjoyment of music, making music an effective medium for exploring their potential. In this study, we explore the parental perception of MT through an online survey. Contrary to the public perception that...

In this paper, we visualize children's coordinated gaze, gesture, and vocalization to better understand communicative behaviors and to identify developmental delay, specifically in the domain of Autism Spectrum Disorders. To date, existing behavioral data from clinical assessment instruments are often stored in raw text files or spreadsheets. This...

Even though electrodermal activity has been widely used in the study of psychological states and processes for over 130 years, the use of such technology in situ, within the context of daily activities, remains a major challenge. Recent technological advancements have led to the development of wearable biosensors that noninvasively measure electric...

Data visualization is by far the most commonly used mechanism to explore and extract insights from datasets, especially by novice data scientists. And yet, current visual analytics tools are rather limited in their ability to operate on collections of visualizations---by composing, filtering, comparing, and sorting them---to find those that depict...

To investigate what kind of snippets are better suited for structured search on mobile devices, we built an experimental mobile search application and conducted a task-oriented interactive user study with 36 participants. Four different versions of a search engine result page (SERP) were compared by varying the snippet type (query-biased vs. non-re...

The outfits people wear contain latent fashion concepts capturing styles, seasons, events, and environments. Fashion theorists have proposed that these concepts are shaped by design elements such as color, material, and silhouette. A dress may be "bohemian" because of its pattern, material, trim, or some combination of them: it is not always clear...

Joint attention is widely recognized as an important developmental milestone for children, and experts consider a lack of joint attention a defining characteristic of autism spectrum disorders (ASDs). While clinicians and researchers agree on the importance of joint attention, their definitions and methods for assessing joint attention vary. In thi...

Traditional medical fundraising charities have been relying on third-party watchdogs and carefully crafting their reputation over time to signal their credibility to potential donors. As medical fundraising campaigns migrate to online platforms in the form of crowdfunding, potential donors can no longer rely on the organization's traditional method...

Many online platforms use curation algorithms that are opaque to the user. Recent work suggests that discovering a filtering algorithm's existence in a curated feed influences user experience, but it remains unclear how users reason about the operation of these algorithms. In this qualitative laboratory study, researchers interviewed a diverse, non...

Data visualization is by far the most commonly used mechanism to explore data, especially by novice data analysts and data scientists. And yet, current visual analytics tools are rather limited in their ability to guide data scientists to interesting or desired visualizations: the process of visual data exploration remains cumbersome and time-consu...

Maps have long played a crucial role in enabling people to conceptualize and navigate the world around them. However, maps also encode the world-views of their creators. Disputed international borders are one example of this: governments may mandate that cartographers produce maps that conform to their view of a territorial dispute. Today, online m...

Computer algorithms organize and select information across a wide range of applications and industries, from search results to social media. Abuses of power by Internet platforms have led to calls for algorithm transparency and regulation. Algorithms have a particularly problematic history of processing information about race. Yet some analysts hav...

Social reminder interfaces on social networking sites (SNSs), such as the Facebook birthday reminder, make sending a congratulatory message easier than ever. However, the lower cost in time and effort can also devalue a simple message, and one-click congratulations may be criticized as impersonal. Nevertheless, they are still widely used. In this p...

Multilingual users of social networking sites (SNSs) write in different languages for various reasons. In this paper, we explore the language choice of multilingual Chinese and Korean students studying in the United States on Facebook. We survey the effects of collectivist culture, imagined audience, and language proficiency on their language choic...

" Answering questions with data is a difficult and time-consuming process. Visual dashboards and templates make it easy to get started, but asking more sophisticated questions often requires learning a tool designed for expert analysts. Natural language interaction allows users to ask questions directly in complex programs without having to learn h...