Margaret M Burnett

Oregon State University | OSU · School of Electrical Engineering and Computer Science

Md Montaser Hamid, Amreeta Chatterjee, Mariam Guizani, Andrew Anderson, Fatima Moussaoui, Sarah Yang, Isaac Tijerina Escobar, Anita Sarma, and Margaret Burnett

Assessing an AI system's behavior-particularly in Explainable AI Systems-is sometimes done empirically, by measuring people's abilities to predict the agent's next move-but how to perform such measurements? In empirical studies with humans, an obvious approach is to frame the task as binary (i.e., prediction is either right or wrong), but this does...

Motivations: Recent research has emerged on generally how to improve AI products’ Human-AI Interaction (HAI) User Experience (UX), but relatively little is known about HAI-UX inclusivity. For example, what kinds of users are supported, and who are left out? What product changes would make it more inclusive? Objectives: To help fill this gap, we pre...

What if “regular” CS faculty each taught elements of inclusive design in “regular” CS courses across an undergraduate curriculum? Would it affect the CS program's climate and inclusiveness to diverse students? Would it improve retention? Would students learn less CS? Would they actually learn any inclusive design? To answer these questions, we cond...

Emerging research shows that individual differences in how people use technology sometimes cluster by socioeconomic status (SES) and that when technology is not socioeconomically inclusive, low-SES individuals may abandon it. To understand how to improve technology's SES-inclusivity, we present a multi-phase case study on SocioEconomicMag (SESMag),...

How can an entire CS faculty, who together have been teaching the ACM standard CS curricula, shift to teaching elements of inclusive design across a 4-year undergraduate CS program? And will they even want to try? To investigate these questions, we developed an educate-the-educators curriculum to support this shift. The overall goal of the educate-...

Would you allow an AI agent to make decisions on your behalf? If the answer is “not always,” the next question becomes “in what circumstances”? Answering this question requires human users to be able to assess an AI agent—and not just with overall pass/fail assessments or statistics. Here users need to be able to localize an agent’s bugs so that th...

Explainable AI is growing in importance as AI pervades modern society, but few have studied how explainable AI can directly support people trying to assess an AI agent. Without a rigorous process, people may approach assessment in ad hoc ways—leading to the possibility of wide variations in assessment of the same agent due only to variations in the...

"In what circumstances would you want this AI to make decisions on your behalf?" We have been investigating how to enable a user of an AI-powered system to answer questions like this through a series of empirical studies, a group of which we summarize here. We began the series by (1) comparing four explanation configurations of saliency explanation...

Although inequities and biases relating to people in low socioeconomic situations are starting to capture widespread attention in the popular press, little attention has been given to how such inequities and biases might pervade technology user experiences. Without understanding such inequities in user experiences, technology designers can unwittin...

Artificial Intelligence (AI) is becoming more pervasive through all levels of society, trying to help us be more productive. Research like Amershi et al.'s 18 guidelines for human-AI interaction aim to provide high-level design advice, yet little remains known about how people react to Applications or Violations of the guidelines. This leaves a gap...

Gender inclusivity in software is gaining attention from researchers and practitioners, with some seeing it as a nonfunctional requirement. To investigate how gender inclusivity can be incorporated into creating software, we gathered data during periods ranging from 5 months to 3.5 years from 10 software teams that used the Gender Inclusiveness Mag...

How should reinforcement learning (RL) agents explain themselves to humans not trained in AI? To gain insights into this question, we conducted a 124-participant, four-treatment experiment to compare participants’ mental models of an RL agent in the context of a simple Real-Time Strategy (RTS) game. The four treatments isolated two types of explana...

Assessing and understanding intelligent agents is a difficult task for users who lack an AI background. ?Explainable AI? (XAI) aims to address this problem, but what should be in an explanation? One route toward answering this question is to turn to theories of how humans try to obtain information they seek. Information Foraging Theory (IFT) is one...

Previous research has revealed that newcomer women are disproportionately affected by gender-biased barriers in open source software (OSS) projects. However, this research has focused mainly on social/cultural factors, neglecting the software tools and infrastructure. To shed light on how OSS tools and infrastructure might factor into OSS barriers...

Recent years have seen a growing number of calls for considering gender in the design or evaluation of software, websites, or other digital technology. Calls like these have arisen from an emerging awareness in HCI of findings from the social sciences that are relevant to the way people use and design technology. However, emerging work on bringing...

We present a user study to investigate the impact of explanations on non-experts? understanding of reinforcement learning (RL) agents. We investigate both a common RL visualization, saliency maps (the focus of attention), and a more recent explanation type, reward-decomposition bars (predictions of future types of rewards). We designed a 124 partic...

Version Control Systems (VCS) are an important source of information for developers. This calls for a principled understanding of developers' information seeking in VCS-both for improving existing tools and for understanding requirements for new tools. Our prior work investigated empirically how and why developers seek information in VCS: in this p...

Although the need for gender-inclusivity in software itself is gaining attention among both SE researchers and SE practitioners, and methods have been published to help, little has been reported on how to make such methods work in real-world settings. For example, how do busy software practitioners use such methods in low-cost ways? How do they end...

Most software systems today do not support cognitive diversity. Further, because of differences in problem-solving styles that cluster by gender, software that poorly supports cognitive diversity can also embed gender biases. To help software professionals fix gender bias "bugs" related to people's problem-solving styles for information processing...

How can software practitioners assess whether their software supports diverse users? Although there are empirical processes that can be used to find "inclusivity bugs" piecemeal, what is often needed is a systematic inspection method to assess soft-ware's support for diverse populations. To help fill this gap, this paper introduces InclusiveMag, a...

In recent years, research has revealed gender biases in numerous software products. But although some researchers have found ways to improve gender participation in specific software projects, general methods focus mainly on detecting gender biases -- not fixing them. To help fill this gap, we investigated whether the GenderMag bias detection metho...

Web-active end-user programmers squander much of their time foraging for bugs and related information in mashup programming environments as well as on the web. To analyze this foraging behavior while debugging, we utilize an Information Foraging Theory perspective. Information Foraging Theory models the human (predator)behavior to forage for specif...

We present a user study to investigate the impact of explanations on non-experts' understanding of reinforcement learning (RL) agents. We investigate both a common RL visualization, saliency maps (the focus of attention), and a more recent explanation type, reward-decomposition bars (predictions of future types of rewards). We designed a 124 partic...

Inclusive design is important in today's software industry, but there is little research about how to teach it. In collaboration with 9 teacher-researchers across 8 U.S. universities and more than 400 computer and information science students, we embarked upon an Action Research investigation to gather insights into the pedagogical content knowledg...

This position paper considers what studying Open Source Software tools can lend to understanding the topic of Gender Diversity in Open Source Software. More specifically we investigate the GenderMag method, a Gender Inclusive method and how it can help increase gender inclusiveness in the tools that are used by OSS communities.

Research has revealed that significant barriers exist when entering Open-Source Software (OSS) communities and that women disproportionately experience such barriers. However, this research has focused mainly on social/cultural factors, ignoring the environment itself --- the tools and infrastructure. To shed some light onto how tools and infrastru...

How should an AI-based explanation system explain an agent's complex behavior to ordinary end users who have no background in AI? Answering this question is an active research area, for if an AI-based explanation system could effectively explain intelligent agents' behavior, it could enable the end users to understand, assess, and appropriately tru...

This panel aims to create a space for participants at CHI 2018 to see how far we have come as a community in raising and addressing issues of gender, and how far we have yet to go. Our intent is for open discussion to support the community's intentions to move towards greater equity, inclusivity, and diversity.

Assessing and understanding intelligent agents is a difficult task for users that lack an AI background. A relatively new area, called "Explainable AI," is emerging to help address this problem, but little is known about how users would forage through information an explanation system might offer. To inform the development of Explainable AI systems...

Assessing and understanding intelligent agents is a difficult task for users that lack an AI background. A relatively new area, called "Explainable AI," is emerging to help address this problem, but little is known about how users would forage through information an explanation system might offer. To inform the development of Explainable AI systems...

How should an AI-based explanation system explain an agent's complex behavior to ordinary end users who have no background in AI? Answering this question is an active research area, for if an AI-based explanation system could effectively explain intelligent agents' behavior, it could enable the end users to understand, assess, and appropriately tru...

One area of research in the end-user development area is known as end-user software engineering (EUSE). Research in EUSE aims to invent new kinds of technologies that collaborate with end users to improve the quality of their software. EUSE has become an active research area since its birth in the early 2000s, with a large body of literature upon w...

Personas often aim to improve product designers' ability to "see through the eyes of" target users through the empathy personas can inspire - but personas are also known to promote stereotyping. This tension can be particularly problematic when personas (who, of course as "people" have genders) are used to promote gender inclusiveness - because rei...

Foraging among similar variants of the same artifact is a common activity, but computational models of Information Foraging Theory (IFT) have not been developed to take such variants into account. Without being able to computationally predict people's foraging behavior with variants, our ability to harness the theory in practical ways--such as buil...

Many systems are designed to help novices who want to learn programming, but few support those who are not necessarily interested in learning programming. This paper targets the subset of end-user programmers (EUPs) in this category. We present a set of principles on how to help EUPs like this learn just a little when they need to overcome a barrie...

Empirical studies have revealed that software developers spend 35%–50% of their time navigating through source code during development activities, yet fundamental questions remain: Are these percentages too high, or simply inherent in the nature of software development? Are there factors that somehow determine a lower bound on how effectively devel...

This short paper is a summary of my keynote at FSE’16, with accompanying references for follow-up.

The design of programming tools is slow and costly. To ease this process, we developed a design pattern catalog aimed at providing guidance for tool designers. This catalog is grounded in Information Foraging Theory (IFT), which empirical studies have shown to be useful for understanding how developers look for information during development tasks....

Intelligent systems are gaining in popularity and receiving increased media attention, but little is known about how people actually go about developing them. In this paper, we attempt to fill this gap through a set of field interviews that investigate how people develop intelligent systems that incorporate machine learning algorithms. The develope...

Recent research has reported numerous studies bringing into question the gender inclusiveness of many kinds of software. Inclusiveness of software (gender or otherwise) matters because supporting diversity matters — it is well-known that the more diverse a group of problem-solvers, the higher the quality of the solution. To help software creators i...

Foraging among too many variants of the same artifact can be problematic when many of these variants are similar. This situation, which is largely overlooked in the literature, is commonplace in several types of creative tasks, one of which is exploratory programming. In this paper, we investigate how novice programmers forage through similar varia...

More people are learning to code than ever, but most learning opportunities do not explicitly teach the problem solving skills necessary to succeed at open-ended programming problems. In this paper, we present a new approach to impart these skills, consisting of: 1) explicit instruction on programming problem solving, which frames coding as a proce...

Programming languages form the interface between programmers (the users) and the computation that they desire the computer to execute. Although studies exist for some aspects of programming language design (such as conditionals), other aspects have received little or no human factors evaluations. Designers thus have little they can rely on if they...

Gender inclusiveness in computing settings is receiving a lot of attention, but one potentially critical factor has mostly been overlooked -- software itself. To help close this gap, we recently created GenderMag, a systematic inspection method to enable software practitioners to evaluate their software for issues of gender-inclusiveness. In this p...

In recent years, research into gender differences has established that individual differences in how people problem-solve often cluster by gender. Research also shows that these differences have direct implications for software that aims to support users' problem-solving activities, and that much of this software is more supportive of problem-solvi...

Many systems are designed to help novices who want to learn programming, but few support those who are not interested in learning (more) programming. This paper targets the subset of end-user programmers (EUPs) in this category. We present a set of principles on how to help EUPs like this learn just a little when they need to overcome a barrier. We...

Developers performing maintenance activities must balance their efforts to learn the code vs. their efforts to actually change it. This balancing act is consistent with the “production bias” that, according to Carroll’s minimalist learning theory, generally affects software users during everyday tasks. This suggests that developers’ focus on effici...

The paradigm of end-user development enables ordinary end-users of computer systems to engage in the modification, extension and even creation of software artifacts. Technology, organization and context are all important aspects that influence end-users’ decisions to engage in end-user development. With this workshop, we invite researchers and prac...

Recent research has shown that some software that is intended to be gender-neutral is not, in fact, equally inclusive to males and females. But little is known about how to design software in a gender-aware fashion, and existing research on gender differences relevant to software design is scattered across at least five different academic fields (e...

How can end users efficiently influence the predictions that machine learning systems make on their behalf? This paper presents Explanatory Debugging, an approach in which the system explains to users how it made each of its predictions, and the user then explains any necessary corrections back to the learning system. We present the principles unde...

This paper summarizes the keynote address on the future of end-user software engineering. We believe the future that we envision has implications for not only end-user software engineering, but also for “classic” software engineering.

A method and system of determining a prioritized list of one or more users related to a given goal obtaining a set of places, determine one or more future places an expert associated with a given goal is predicted to visit to accomplish the given goal, obtain a history of one or more places users have visited, determine one or more historical place...

Although there are many systems designed to engage people in programming, few explicitly teach the subject, expecting learners to acquire the necessary skills on their own as they create programs from scratch. We present a principled approach to teach programming using a debugging game called Gidget, which was created using a unique set of seven de...

End-user software engineering (EUSE) is a research area that aims to invent new kinds of technologies that collaborate with end users to improve the quality of their software. The practice that EUSE research aims to support is end users using new tools and methods to improve the quality of the software that they and other end users have created. Th...

Although there have been many advances in end-user programming environments, recent empirical studies report that programming still remains difficult for end-users. We hypothesize that one reason may be lack of effective support for helping end-user programmers problem-solve their own way around barriers they encounter. Therefore, in this paper, we...

Interactive technologies have a profound mediating effect on the way we obtain and contribute to knowledge, relate to each other and contribute to society. Often, "gender" is not a factor that is explicitly considered in the design of these technologies. When gender is considered, products are often designed with idealised models of gendered "users...

How do you test a program when only a single user, with no expertise in software testing, is able to determine if the program is performing correctly? Such programs are common today in the form of machine-learned classifiers. We consider the problem of testing this common kind of machine-generated program when the only oracle is an end user: e.g.,...

When intelligent interfaces, such as intelligent desktop assistants, email classifiers, and recommender systems, customize themselves to a particular end user, such customizations can decrease productivity and increase frustration due to inaccurate predictions—especially in early stages when training data is limited. The end user can improve the le...

Research is emerging on how end users can correct mistakes their intelligent agents make, but before users can correctly "debug" an intelligent agent, they need some degree of understanding of how it works. In this paper we consider ways intelligent agents should explain themselves to end users, especially focusing on how the soundness and complete...

End-user programmers often get stuck because they do not know how to overcome their barriers. We have previously presented an approach called the Idea Garden, which makes minimalist, on-demand problem-solving support available to end-user programmers in trouble. Its goal is to encourage end users to help themselves learn how to overcome programming...

Technology has a profound mediating effect on the way we relate, obtain knowledge, and contribute to society. Worldwide there is a gender gap in technology with only a small part of all computer science related positions being held by women. Given the impact and potential ramifications of technology on our society, it is imperative that both male a...