ArticlePDF Available

Avoiding an Oppressive Future of Machine Learning: A Design Theory for Emancipatory Assistants

December 2020
MIS Quarterly

DOI:10.25300/MISQ/2021/1578

Authors:

Boston College

University of Arkansas at Fayetteville

University of Southern California

Boston College

Widespread use of machine learning (ML) systems could result in an oppressive future of ubiquitous monitoring and behavior control that, for dialogic purposes, we call "Informania." This dystopian future results from ML systems' inherent design based on training data rather than built with code. To avoid this oppressive future, we develop the concept of an emancipatory assistant (EA), an ML system that engages with human users to help them understand and enact emancipatory outcomes amidst the oppressive environment of Informania. Using emancipatory pedagogy as a kernel theory, we develop two sets of design principles: one for the near future and the other for the far-term future. Designers optimize EA on emancipatory outcomes for an individual user, which protects the user from Informania's oppression by engaging in an adversarial relationship with its oppressive ML platforms when necessary. The principles should encourage IS researchers to enlarge the range of possibilities for responding to the influx of ML systems. Given the fusion of social and technical expertise that IS research embodies, we encourage other IS researchers to theorize boldly about the long-term consequences of emerging technologies on society and potentially change their trajectory.

A Basic Machine Learning Model

The future of HMLP in the workplace without emancipatory organizing processes.

Oppressive Features of the Basic ML Model

The future of HMLP in the workplace if the 4 emancipatory organizing processes (EOPs) are adopted.

Contextualizing Emancipatory Pedagogy Modifies the Basic ML Model

Figures - uploaded by Amber Young

Content may be subject to copyright.

Content uploaded by Amber Young

Content may be subject to copyright.

Content uploaded by Amber Young

Content may be subject to copyright.

Content uploaded by Amber Young

Content may be subject to copyright.

Content uploaded by Amber Young

Content may be subject to copyright.

Avoiding an Oppressive

Future of Machine

Learning: A Design Theory

for Emancipatory

Assistants

Kane, G. C., Young, A. G., Majchrzak, A., & Ransbotham,

S. (2021). Avoiding an Oppressive Future of Machine

Learning: A Design Theory for Emancipatory Assistants.

MIS Quarterly, 45(1), pp. 371-396.





•



•



•



•  ! 



•Informania"

#



!$

$!





•  !

%





& 





• '

•$

•(



)

%



•“A problem well stated is half

solved” *+,

•+% 



 



• 

(

•



*$





- ."/

 .



•Emancipation 

#

• !oppression#

%

•  !





•- .$



)

0%



1

#



2



3$

&



•)

!(



•450"

%

6

•451" 

$!



•452"4

$

7

•453".



$





•)design theory 





•



•)$





-



$

•$

6





$



• 





8



9



-:

-:$







•0"+

%



•1":



$

•2"+





•3"/

$

0"





•:-:









•:-:





•:-:

9



1"

%

;

•:-:;

8;

 

•:-:;

 

•:-:

%

2"

%



•:-:

9

•:-:



•:-:



•:-:





3"







•-:



•-:





•-:





4%



-<-:

•-

-:

•+

•





• %!





 

•$

•-:

• %9(

=(

•)>.2!

.$3?"

"@@@#

@

•"

"@@$@@



•A!:(%$!:B,!C1D104%

$

$!International Journal of Information

Management

Avoiding virtual dystopia: A design theory for emancipatory participatory immersive platforms

Article

Full-text available

Jan 2025
J STRATEGIC INF SYST

Immersive platforms, accessed via VR and AR interfaces, offer a profound digital experience but raise significant privacy and ethical concerns. Current systems collect extensive user data; this enables manipulative advertising and behavior control, fosters self-censorship, and diminishes authentic self-expression – especially among marginalized communities. The unchecked development of immersive platforms threatens to create oppressive environments due to power imbalances and a lack of regulation. This paper addresses the urgent need for emancipatory design guidelines to prevent such outcomes. By integrating critical theory with Participatory Design, we propose Emancipatory Participatory Immersive Platforms. These platforms involve users in their design and dismantle oppressive structures through four emancipatory tools (i.e., agency, dialogue, inclusion, and rationality), substantiated by eight actionable design principles. Our framework aims to empower users, promote inclusivity, and ensure rational engagement – and thereby foster immersive platforms that maintain functional benefits while enhancing freedom and equity.

Generative AI and its Transformative Value for Digital Platforms

Article

Full-text available

Apr 2025
J MANAGE INFORM SYST

The emergence of generative artificial intelligence (GenAI) represents a watershed moment in the evolution of digital platforms. The capabilities of this AI technology go beyond traditional AI systems, enabling the autonomous generation of novel outcomes with significant implications for platform value creation, architecture, govern-ance, and stakeholder interactions. We develop an integrative conceptual framework that identifies four key mechanisms through which GenAI transforms digital platforms: intelligent automation, democratization, hyper-personalization, and collaborative innovation. Through intelligent automation, GenAI transforms boundary resources from passive interfaces into active, intelligent mediators of value creation. Democratization systematically lowers barriers to platform participation. Hyper-personalization enables dynamic, individual-level adaptation of platform content. Collaborative innovation transforms platform innovation by making GenAI an active participant in human-AI value co-creation. We use this framework to situate the papers in the special issue and develop a research agenda that explores the transformative impact of GenAI on platform stakeholder relationships.

Search and Society: Reimagining Information Access for Radical Futures

Article

Full-text available

Mar 2025

Bhaskar Mitra

Information retrieval (IR) technologies and research are undergoing transformative changes. It is our perspective that the community should accept this opportunity to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as social and political sciences, and should be co-developed with cross-disciplinary scholars, legal and policy experts, civil rights and social justice activists, and artists, among others. In this perspective paper, we motivate why the community must consider this radical shift in how we do research and what we work on, and sketch a path forward towards this transformation.

Make Up Your Mind! Habit Engineering With Artificial Intelligence in the Context of Trading

Conference Paper

Jun 2025

Habits play a critical role in decision-making, affecting both our personal and professional lives. While good habits improve efficiency and decision quality, bad habits can reinforce biases and hinder adaptability. Artificial intelligence (AI) offers the potential of habit engineering: it can promote beneficial behavioral changes. This study examines how AI influences the habits of decision-makers and how organizations can use AI to foster effective routines. Through a case study at Allianz Global Investors (AGI), we analyze the impact of AI-driven advice on traders’ decision-making processes in a high-stakes environment. The results highlight AI’s ability to disrupt established habits, encouraging greater reflection and improved performance in trading decisions. This research contributes to the literature on habit engineering and organizational AI by highlighting the need for careful monitoring and evaluation of AI systems to balance efficiency and adaptability, ensuring that habits remain aligned with organizational goals and long-term success.

Visibility vs. virality and from hashtags to hysteria: how orthopaedic surgeons can surf the social media tsunami without drowning?

Article

May 2025
INT ORTHOP

Technology Advancements Shaping the Financial Inclusion Landscape: Present Interventions, Emergence of Artificial Intelligence and Future Directions

Article

Full-text available

Mar 2025
INFORM SYST FRONT

The global commitment to advancing financial inclusion (FI) relies on technology to connect underserved communities with the formal financial sector. Existing traditional technologies have made some progress, but they often fail to adapt to the unique needs of these populations. Although artificial intelligence (AI) offers new possibilities to meet these limitations, its rapid advancement has outpaced the development of integrative studies, leaving its potential impact on financial access by the underserved largely unexplored. Existing research provides fragmented insights into how different technological interventions impact diverse groups. We develop a segment-outcome-focused analysis to structure a scoping review of 95 information systems studies to assess current technological advances for FI and explore how AI can address their limitations, including limited digital literacy, uneven and high cost of infrastructure, and service personalization. We then engender future research directions and conclude with theoretical contributions and practical implications, emphasizing the potential of AI solutions to advance FI.

The Ethics of Data and Its Governance: A Discourse Theoretical Approach

Article

Jun 2025

Bernd Carsten Stahl

The rapidly growing amount and importance of data across all aspects of organisations and society have led to urgent calls for better, more comprehensive and applicable approaches to data governance. One key driver of this is the use of data in machine learning systems, which hold the promise of producing much social and economic good, but which simultaneously raise significant concerns. Calls for data governance thus typically have an ethical component. This can refer to specific ethical values that data governance is meant to preserve, most obviously in the area of privacy and data protection. More broadly, responsible data governance is seen as a condition of the development and use of ethical and trustworthy digital technologies. This conceptual paper takes the already existing ethical aspect of the data governance discourse as a point of departure and argues that ethics should play a more central role in data governance. Drawing on Habermas’s Theory of Communicative Action and using the example of neuro data, this paper argues that data shapes and is shaped by discourses. Data is at the core of our shared ontological positions and influences what we believe to be real and thus also what it means to be ethical. These insights can be used to develop guidance for the further development of responsible data governance.

AI and the metaverse in the workplace: DEI opportunities and challenges

Article

Feb 2024
PERS REV

Purpose The metaverse, through artificial intelligence (AI) systems and capabilities, allows considerable data analysis in the workplace, largely exceeding traditional people analytics data collection. While concerns over surveillance and issues associated with privacy and discrimination have been raised, the metaverse has the potential to offer opportunities associated with fairer assessment of employee performance and enhancement of the employee experience, especially with respect to gender and race, inclusiveness and workplace equity. This paper aims at shedding light on the diversity, equity and inclusion (DEI) opportunities and challenges of implementing the metaverse in the workplace, and the role played by AI. Design/methodology/approach This paper draws on our past research on AI and the metaverse and provides insights addressed to human resources (HR) scholars and practitioners. Findings Our analysis of AI applications to the metaverse in the workplace sheds light on the ambivalent role of and potential trade-offs that may arise with this emerging technology. If used responsibly, the metaverse can enable positive changes concerning the future of work, which can promote DEI. Yet, the same technology can lead to negative DEI outcomes if implementations occur quickly, unsupervised and with a sole focus on efficiencies and productivity (i.e. collecting metrics, models etc.). Practical implications Managers and HR leaders should try to be first movers rather than followers when deciding if (or, better, when) to implement metaverse capabilities in their organizations. But how the metaverse is implemented will be strategic. This involves choices concerning the degree of invasive/pervasive monitoring (internal) as well as make or buy decisions concerning outsourcing AI capabilities. Originality/value Our paper is one among few (to date) that discusses AI capabilities in the metaverse at the intersection of the HR and information systems(IS) literature and that specifically tackles DEI issues. Also, we take a “balanced” approach when evaluating the metaverse from a DEI perspective. While most studies either demonize or celebrate these technologies from an ethical and DEI standpoint, we aim to highlight challenges and opportunities, with the goal to guide scholars and practitioners towards a responsible use of the metaverse in organizations.

Using sustainable technology to drive efficiency: Artificial intelligence as an information broker for advancing airline operations management

Article

May 2025

Supporting Indigenous Expressions of Gender Fluidity and Sexual Diversity Through ICT: An Examination of Two-Spirit Web Pages

Article

Apr 2025

Jose Ortiz

Websites are one of the most prolific forms of information and communication technology (ICT). Yet, their potential to contribute to the development of oppressed groups and marginalized populations remains largely understudied. The ICT for development literature often uses simplistic views of identity that ignore the diversity of human experiences arising from the interconnectedness of ethnicity, sexual orientation, and gender. These aspects of identity are interconnected, and ignoring their intersection disregards compounding inequalities contended by sexual and gender minorities who are also ethnic minorities. Motivated by this research gap, this study analyzes 33 Two-Spirit web pages to further our understanding of how websites can contribute to the development of Two-Spirit communities. Two-Spirit is a modern term that recognizes expressions of gender fluidity and sexual diversity advanced by Indigenous Peoples in North America for centuries. An interpretive approach revealed that websites can support the development of Two-Spirit communities through four affordances: open spaces for activism, life stories telling, initiative impact reporting, and enabling access to resources. This paper contributes to the ICT literature on development by explaining how these affordances coalesce into a visibilization mechanism whereby ICT mediates unconstrained expressions of the Two-Spirit identity free of prejudices and colonial distortion.

Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach

Chapter

Full-text available

Oct 2020
Lect Notes Comput Sci

Explanations—a form of post-hoc interpretability—play an instrumental role in making systems accessible as AI continues to proliferate complex and sensitive sociotechnical systems. In this paper, we introduce Human-centered Explainable AI (HCXAI) as an approach that puts the human at the center of technology design. It develops a holistic understanding of “who” the human is by considering the interplay of values, interpersonal dynamics, and the socially situated nature of AI systems. In particular, we advocate for a reflective sociotechnical approach. We illustrate HCXAI through a case study of an explanation system for non-technical end-users that shows how technical advancements and the understanding of human factors co-evolve. Building on the case study, we lay out open research questions pertaining to further refining our understanding of “who” the human is and extending beyond 1-to-1 human-computer interactions. Finally, we propose that a reflective HCXAI paradigm—mediated through the perspective of Critical Technical Practice and supplemented with strategies from HCI, such as value-sensitive design and participatory design—not only helps us understand our intellectual blind spots, but it can also open up new design and research spaces.

Emancipation Research in Information Systems: Integrating Agency, Dialogue, Inclusion, and Rationality Research

Conference Paper

Full-text available

Jan 2021

Emancipation is a key concept in critical theories. Prior work suggests that emancipation is a complex and multi-faceted concept. Many conceptualizations of emancipation exist, and emancipation is defined in different ways. Existing empirical studies mainly focus on one or few components of emancipation. To have an integrated understanding of emancipation, we review the literature on emancipation in information systems (IS), with a view toward developing a typology of components of emancipation in the IS field. The typology of emancipation components consists of four components: freedom to act, freedom to express, freedom to belong and freedom to think. These components relate to the concepts of agency, dialogue, inclusion, and rationality, respectively.

Environmental Education in Brazil and the Influence of Paulo Freire Summary and Keywords

Article

Full-text available

May 2020

Marcos Reigota

Studying the Other or Becoming the Other: Engaging with Indigenous Peoples in IS Research

Article

Full-text available

Apr 2020

This article is based on a panel discussion at the 2019 International Conference on Information Systems (ICIS) held in Munich, Germany. This panel was concerned with the ethics and politics of engagement with Indigenous peoples in information systems research. As members of a research team that have been studying the use of social media by Indigenous peoples to collaborate and further their cause, we have recently become aware of some of the unintended consequences of IS research. Since others could easily appropriate our findings for political purposes, we believe that we as IS researchers need to become more sensitive to the ways in which we study and engage with “the Other.” Hence, the panelists discussed and debated the nature and extent of a researcher’s engagement when studying Indigenous peoples and their uses of IS/IT. The panel, chaired by Michael Myers, included three panelists who have been studying Indigenous peoples’ use of social media (Liz Davidson, Amber Young and Hameed Chughtai), and one panelist who is an Indigenous scholar studying Indigenous theories in IS (Pitso Tsibolane).

The rise of human machines: How cognitive computing systems challenge assumptions of user-system interaction

Article

Full-text available

Feb 2020
J ASSOC INF SYST

Cognitive computing systems (CCS) are a new class of computing systems that implement more human-like cognitive abilities. CCS are not a typical technological advancement but an unprecedented advance toward human-like systems fueled by artificial intelligence. Such systems can adapt to situations, perceive their environments, and interact with humans and other technologies. Due to these properties, CCS are already disrupting established industries, such as retail, insurance and healthcare. As we make the case in this paper, the increasingly human-like capabilities of CCS challenge five fundamental assumptions that we—as IS researchers—have held about how users interact with IT artifacts. These assumptions pertain to the (i) direction of the user-artifact relationship, (ii) artifact’s awareness of its environment, (iii) functional transparency, (iv) reliability, and (v) user’s awareness of artifact use. We argue that the disruption of these five assumptions limit the applicability of our extant body of knowledge to CCS. Consequently, CCS present a unique opportunity for novel theory development and associated contributions. We argue that IS is well positioned to take this opportunity and present research questions that, if answered, will lead to interesting, influential and original theories.

Counter-Arguing During Curriculum-Supported Peer Interaction Facilitates Middle-School Students’ Science Content Knowledge

Article

Full-text available

Jul 2019

Peer argumentation, especially the discussion of contrary points of view, has experimentally been found to be effective in promoting science content knowledge, but how this occurs is still unknown. The available explanations are insufficient because they do not account for the evidence showing that gains in content knowledge are unrelated to group outcomes and are still evident weeks after collaboration occurs. The aim of this article is to contribute to the understanding of the relationship between peer-group argumentation and science content knowledge learning. A total of 187 students (aged 10 to 11 years) from 8 classrooms participated in the study, with the classrooms spread across 8 public schools, all located in Santiago, Chile. We conducted a quasi-experimental study randomized at school-class level. Four teachers delivered science lessons following a teaching program especially developed to foster dialogic and argumentative classroom talk (the intervention group), and four teachers delivered lessons in their usual way (the control group). Students were assessed individually using both immediate and delayed post-test measures of science content knowledge. The results showed no differences in pre- to post-immediate content knowledge between conditions. However, the intervention-group students increased their content knowledge significantly more than the control-group students between post-immediate and post-delayed tests. Hierarchical multiple regression analyses showed that, after controlling for school-level variables, time working in groups, and scores in the pretest, the formulation of counter arguments, although occurring in both groups, significantly predicted delayed gains in the intervention group only. Moreover, the frequency of counterarguments heard by students during the group work did not make a difference. Focal analysis of one small-group work suggests that teachers’ instructional practice may have contributed to the consolidation of students’ knowledge at an individual level in a post-collaborative phase.

The Age of Surveillance Capitalism

Chapter

Apr 2023

Shoshana Zuboff

The Technology Fallacy: How People Are the Real Key to Digital Transformation

Book

Apr 2019

Why an organization's response to digital disruption should focus on people and processes and not necessarily on technology. Digital technologies are disrupting organizations of every size and shape, leaving managers scrambling to find a technology fix that will help their organizations compete. This book offers managers and business leaders a guide for surviving digital disruptions—but it is not a book about technology. It is about the organizational changes required to harness the power of technology. The authors argue that digital disruption is primarily about people and that effective digital transformation involves changes to organizational dynamics and how work gets done. A focus only on selecting and implementing the right digital technologies is not likely to lead to success. The best way to respond to digital disruption is by changing the company culture to be more agile, risk tolerant, and experimental. The authors draw on four years of research, conducted in partnership with MIT Sloan Management Review and Deloitte, surveying more than 16,000 people and conducting interviews with managers at such companies as Walmart, Google, and Salesforce. They introduce the concept of digital maturity—the ability to take advantage of opportunities offered by the new technology—and address the specifics of digital transformation, including cultivating a digital environment, enabling intentional collaboration, and fostering an experimental mindset. Every organization needs to understand its “digital DNA” in order to stop “doing digital” and start “being digital.” Digital disruption won't end anytime soon; the average worker will probably experience numerous waves of disruption during the course of a career. The insights offered by The Technology Fallacy will hold true through them all. A book in the Management on the Cutting Edge series, published in cooperation with MIT Sloan Management Review.

An Approach to Labeling Audio Tags Based on Self-Attention Generative Adversarial Networks

Conference Paper

Dec 2019

Specialized Information Systems for the DigitallyDisadvantaged

Article

Jan 2019
J ASSOC INF SYST

Conference Paper

Full-text available

Emancipation Research in Information Systems: Integrating Agency, Dialogue, Inclusion, and Rationali...

January 2021

Emancipation is a key concept in critical theories. Prior work suggests that emancipation is a complex and multi-faceted concept. Many conceptualizations of emancipation exist, and emancipation is defined in different ways. Existing empirical studies mainly focus on one or few components of emancipation. To have an integrated understanding of emancipation, we review the literature on emancipation ... [Show full abstract] in information systems (IS), with a view toward developing a typology of components of emancipation in the IS field. The typology of emancipation components consists of four components: freedom to act, freedom to express, freedom to belong and freedom to think. These components relate to the concepts of agency, dialogue, inclusion, and rationality, respectively.

Article

Full-text available

Reflection on Writing a Theory Paper: How to Theorize for the Future

September 2021 · Journal of the Association for Information Systems

This manuscript describes our experiences writing a future-oriented, next-generation theory. We provide advice for other theory authors and for reviewers of theory papers. Our advice relates to theorizing challenges as well as the practical and logistical challenges associated with publishing a theory paper. We conclude by calling attention to areas that warrant greater attention from IS ... [Show full abstract] researchers and by describing opportunities for future-oriented theoretical and empirical research. We explain how future-oriented approaches to speculative design and radical reimagining can inform new IS theories. Future-oriented approaches to forecasting and scenario-based research can be leveraged in empirical studies. We hope this editorial will inspire others to theorize and conduct research with an eye to the future. We argue that successful future-oriented research will consider possible futures that are high-likelihood and/or high-risk/reward, and will equip the IS discipline going forward, whatever future may arise.

Conference Paper

Full-text available

"Ideal Speech" on Wikipedia: Balancing Social Marginalization Risks and Social Inclusion Benefits fo...

January 2021

On Wikipedia, editors use talk pages to debate whether/how to include potentially marginalizing associations in highly visible digital information. Despite efforts to promote social inclusion, digitally cocreated information on Wikipedia is vulnerable to marginalizing content. We study the case of Tim Cook's Wikipedia biography prior to him coming out as a member of the LGBTQ community. Editors ... [Show full abstract] cocreating his biography discussed at length whether/how to include information about Cook's sexual orientation. Our critical hermeneutic investigation of these discussions reveals a paradox of social inclusion. That is, efforts by activist editors to promote social inclusion at the group level may bring about the unintended consequence of marginalizing an individual. Applying critical social theories, we conclude that deference should be given to individuals over groups when collaborative decisions are made about whether to publish marginalizing associations in highly visible digital information. This research highlights the complexity of governance for social inclusion online.

Article

Full-text available

Organizing workers and machine learning tools for a less oppressive workplace

August 2021 · International Journal of Information Management

Machine learning tools are increasingly infiltrating everyday work life with implications for workers. By looking at machine learning tools as part of a sociotechnical system, we explore how machine learning tools enforce oppression of workers. We theorize, normatively, that with reorganizing processes in place, oppressive characteristics could be converted to emancipatory characteristics. ... [Show full abstract] Drawing on Paulo Freire’s critical theory of emancipatory pedagogy, we outline similarities between the characteristics Freire saw in oppressive societies and the characteristics of currently designed partnerships between humans and machine learning tools. Freire’s theory offers a way forward in reorganizing humans and machine learning tools in the workplace. Rather than advocating human control or the decoupling of workers and machines, we follow Freire’s theory in proposing four processes for emancipatory organizing of human and machine learning partnership: 1) awakening of a critical consciousness, 2) enabling role freedom, 3) instituting incentives and sanctions for accountability, and 4) identifying alternative emancipatory futures. Theoretical and practical implications of this emancipatory organizing theory are drawn.