No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Start Generating: Harnessing Generative Artificial Intelligence for Sociological Research
Abstract
How can generative artificial intelligence (GAI) be used for sociological research? The author explores applications to the study of text and images across multiple domains, including computational, qualitative, and experimental research. Drawing upon recent research and stylized experiments with DALL·E and GPT-4, the author illustrates the potential applications of text-to-text, image-to-text, and text-to-image models for sociological research. Across these areas, GAI can make advanced computational methods more efficient, flexible, and accessible. The author also emphasizes several challenges raised by these technologies, including interpretability, transparency, reliability, reproducibility, ethics, and privacy, as well as the implications of bias and bias mitigation efforts and the trade-offs between proprietary models and open-source alternatives. When used with care, these technologies can help advance many different areas of sociological methodology, complementing and enhancing our existing toolkits.
... The current article is not the rst exploration of how to use generative AI in higher education research [22]; many good publications exist [23][24][25]. The current article adds a new approach by demonstrating the application of generative AI in Mixed Methods Research (MMR) and providing prompts and guidelines for use. ...
... Unfortunately, he is a people-pleaser, which can sometimes be annoying. ChatGPT and similar generative AI platforms can analyse multimodal data [23] and play several roles in qualitative analysis, including coding, where you ask ChatGPT to generate codes for textual data based on frameworks or theories. In addition to textual analysis, ChatGPT can detect sentiment, such as the emotional tone of the text [79]. ...
The current article used real data to demonstrate the analysis and synthesis of Mixed Methods Research (MMR) data with generative Artificial Intelligence (Gen AI). I explore how reliable and valid Gen AI data outputs are and how to improve their use. The current content is geared towards enhancing methodological application regardless of field or discipline and includes access to a prompt library and examples of using outputs. The demonstration data used emanated from a study done in South Africa, with a quantitative sample size of 969 first-year engineering students and, for the qualitative part, 14 second-year students. In the current article, I compare my original analysis to ChatGPT results. Generative AI as a mind tool is best used with human insight, and I found this to be especially true when coding qualitative data. ChatGPT produced generic codes if asked to do inductive coding, and the results improved when training the Gen AI on human examples, which led to moderate and significant correlations between human and machine coding. The quantitative analysis was accurate for the descriptive statistics, but the researcher had to use best judgment to select the correct inferential analysis. Quantitative and qualitative analysis should be conducted separately in generative AI before asking the Chatbot for help with mixed methods results. In the current paper, I give guidelines and a tutorial on how to use chatbots in an ethically responsible and scientifically sound manner for research in social and human sciences.
... LLMs are being rapidly adopted for a wide range of uses among scholars, including document annotation [28,82], information extraction [21], and tool development with applications to healthcare, education, and finance [48,58,71,87]. Social scientists have also been cautiously optimistic about potential uses for LLMs in research [23,42]. The breadth of these use cases make it imperative to best understand their stylistic tendencies for language with respect to various social and scientific contexts. ...
Large language models have become popular over a short period of time because they can generate text that resembles human writing across various domains and tasks. The popularity and breadth of use also put this technology in the position to fundamentally reshape how written language is perceived and evaluated. It is also the case that spoken language has long played a role in maintaining power and hegemony in society, especially through ideas of social identity and “correct” forms of language. But as human communication becomes even more reliant on text and writing, it is important to understand how these processes might shift and who is more likely to see their writing styles reflected back at them through modern AI. We therefore ask the following question: who does generative AI write like? To answer this, we compare writing style features in over 150,000 college admissions essays submitted to a large public university system and an engineering program at an elite private university with a corpus of over 25,000 essays generated with GPT-3.5 and GPT-4 to the same writing prompts. We find that human-authored essays exhibit more variability across various individual writing style features (e.g., verb usage) than AI-generated essays. Overall, we find that the AI-generated essays are most similar to essays authored by students who are males with higher levels of social privilege. These findings demonstrate critical misalignments between human and AI authorship characteristics, which may affect the evaluation of writing and calls for research on control strategies to improve alignment.
... ChatGPT and similar generative AI platforms can analyse multimodal data [23] and play several roles in qualitative analysis, including coding, where you ask ChatGPT to generate codes for textual data based on frameworks or theories. In addition to textual analysis, Gen AIs can detect sentiment, such as the emotional tone of the text [79]. ...
The current article used real data to demonstrate the analysis and synthesis of Mixed Methods Research (MMR) data with generative Artificial Intelligence (Gen AI). I explore how reliable and valid Gen AI data outputs are and how to improve their use. The current content is geared towards enhancing methodological application regardless of field or discipline and includes access to a prompt library and examples of using outputs. The demonstration data used emanated from a study done in South Africa, with a quantitative sample size of 969 first-year engineering students and, for the qualitative part, 14 first-year students. In the current article, I compare my original analysis to ChatGPT results. Generative AI as a mind tool is best used with human insight, and I found this to be especially true when coding qualitative data. ChatGPT produced generic codes if asked to do inductive coding, and the results improved when training the Gen AI on human examples, which led to moderate and significant correlations between human and machine coding. The quantitative analysis was accurate for the descriptive statistics, but the researcher had to use best judgment to select the correct inferential analysis. Quantitative and qualitative analysis should be conducted separately in generative AI before asking the Chatbot for help with mixed methods results. In the current paper, I give guidelines and a tutorial on how to use chatbots in an ethically responsible and scientifically sound manner for research in social and human sciences.
Generative AI that can produce realistic text, images, and other human-like outputs is currently transforming many different industries. Yet it is not yet known how such tools might influence social science research. I argue Generative AI has the potential to improve survey research, online experiments, automated content analyses, agent-based models, and other techniques commonly used to study human behavior. In the second section of this article, I discuss the many limitations of Generative. I examine how bias in the data used to train these tools can negatively impact social science research—as well as a range of other challenges related to ethics, replication, environmental impact, and the proliferation of low-quality research. I conclude by arguing that social scientists can address many of these limitations by creating open-source infrastructure for research on human behavior. Such infrastructure is not only necessary to ensure broad access to high-quality research tools, I argue, but also because the progress of AI will require deeper understanding of the social forces that guide human behavior.
Large Language Models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the Computational Social Science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references.We conclude that the performance of today’s LLMs can augment the CSS research pipeline in two ways: (1) serving as zeroshot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.
There have been widespread concerns about two aspects of the current explosion of predictive text models and other algorithm-based computational tools. On one hand, it is often insisted that Artificial Intelligence (AI) should be made “ethical”, and software providers take this seriously, attempting to make sure that their tools are not used to facilitate grossly criminal or widely condemned activities. On the other hand, it is also widely understood that those who create these tools have a responsibility to ensure that they are “unbiased”, as opposed to simply helping one side in political contestation define their perspectives as reality for all. Unfortunately, these two goals cannot be jointly satisfied, as there are perhaps no ethical prescriptions worthy of notice that are not contested by some. Here I investigate the current ethico-political sensibility of ChatGPT, demonstrating that the very attempt to give it an ethical keel has also given it a measurably left position in the political space and a concomitant position in social space among the privileged.
Careful bias management and data fidelity are key.
We propose and explore the possibility that language models can be studied as effective proxies for specific human subpopulations in social science research. Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models. We show that the “algorithmic bias” within one such tool—the GPT-3 language model—is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups. We term this property algorithmic fidelity and explore its extent in GPT-3. We create “silicon samples” by conditioning the model on thousands of sociodemographic backstories from real human participants in multiple large surveys conducted in the United States. We then compare the silicon and human samples to demonstrate that the information contained in GPT-3 goes far beyond surface similarity. It is nuanced, multifaceted, and reflects the complex interplay between ideas, attitudes, and sociocultural context that characterize human attitudes. We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.
Where do prescient ideas—those that initially challenge conventional assumptions but later achieve widespread acceptance—come from? Although their outcomes in the form of technical innovation are readily observed, the underlying ideas that eventually change the world are often obscured. Here we develop a novel method that uses deep learning to unearth the markers of prescient ideas from the language used by individuals and groups. Our language-based measure identifies prescient actors and documents that prevailing methods would fail to detect. Applying our model to corpora spanning the disparate worlds of politics, law, and business, we demonstrate that it reliably detects prescient ideas in each domain. Moreover, counter to many prevailing intuitions, prescient ideas emanate from each domain's periphery rather than its core. These findings suggest that the propensity to generate far-sighted ideas may be as much a property of contexts as of individuals.
Previous research on emotional language relied heavily on off-the-shelf sentiment dictionaries that focus on negative and positive tone. These dictionaries are often tailored to nonpolitical domains and use bag-of-words approaches which come with a series of disadvantages. This paper creates, validates, and compares the performance of (1) a novel emotional dictionary specifically for political text, (2) locally trained word embedding models combined with simple neural network classifiers, and (3) transformer-based models which overcome limitations of the dictionary approach. All tools can measure emotional appeals associated with eight discrete emotions. The different approaches are validated on different sets of crowd-coded sentences. Encouragingly, the results highlight the strengths of novel transformer-based models, which come with easily available pretrained language models. Furthermore, all customized approaches outperform widely used off-the-shelf dictionaries in measuring emotional language in German political discourse.
The richness of social media data has opened a new avenue for social science research to gain insights into human behaviors and experiences. In particular, emerging data-driven approaches relying on topic models provide entirely new perspectives on interpreting social phenomena. However, the short, text-heavy, and unstructured nature of social media content often leads to methodological challenges in both data collection and analysis. In order to bridge the developing field of computational science and empirical social research, this study aims to evaluate the performance of four topic modeling techniques; namely latent Dirichlet allocation (LDA), non-negative matrix factorization (NMF), Top2Vec, and BERTopic. In view of the interplay between human relations and digital media, this research takes Twitter posts as the reference point and assesses the performance of different algorithms concerning their strengths and weaknesses in a social science context. Based on certain details during the analytical procedures and on quality issues, this research sheds light on the efficacy of using BERTopic and NMF to analyze Twitter data.
The last decade has yielded dramatic and quite surprising breakthroughs in natural language processing through the use of simple artificial neural network computations, replicated on a very large scale and trained over exceedingly large amounts of data. The resulting pretrained language models, such as BERT and GPT-3, have provided a powerful universal language understanding and generation base, which can easily be adapted to many understanding, writing, and reasoning tasks. These models show the first inklings of a more general form of artificial intelligence, which may lead to powerful foundation models in domains of sensory experience beyond just language.
Artificial intelligence (AI)–synthesized text, audio, image, and video are being weaponized for the purposes of nonconsensual intimate imagery, financial fraud, and disinformation campaigns. Our evaluation of the photorealism of AI-synthesized faces indicates that synthesis engines have passed through the uncanny valley and are capable of creating faces that are indistinguishable—and more trustworthy—than real faces.
Sociologists have argued that there is value in incorporating computational tools into qualitative research, including using machine learning to code qualitative data. Yet standard computational approaches do not neatly align with traditional qualitative practices. The authors introduce a hybrid human-machine learning approach (HHMLA) that combines a contemporary iterative approach to qualitative coding with advanced word embedding models that allow contextual interpretation beyond what can be reliably accomplished with conventional computational approaches. The results, drawn from an analysis of 87 human-coded ethnographic interview transcripts, demonstrate that HHMLA can code data sets at a fraction of the effort of human-only strategies, saving hundreds of hours labor in even modestly sized qualitative studies, while improving coding reliability. The authors conclude that HHMLA may provide a promising model for coding data sets where human-only coding would be logistically prohibitive but conventional computational approaches would be inadequate given qualitative foci.
Building on the definition offered by Aspers and Corte, I argue that qualitative research is not qualitative simply because it encodes for the ability “to get closer” to the phenomenon being studied, so much as it is anchored by a methodological obligation to critically examine how and why that closeness matters. Qualitative research considers the positionality of both the researcher and the researched as core aspects of inquiry to understand how knowledge and experience are situated, co-constructed, and historically and socially located. This methodological expectation for reflexivity does not just allow for richer data, but also requires researchers to consider power within and surrounding the research process and to employ an ethic of care for their subjects and for the overall work of qualitative research.
Computational methods have become widespread in the social sciences, but probabilistic language models remain relatively underused. We introduce language models to a general social science readership. First, we offer an accessible explanation of language models, detailing how they estimate the probability of a piece of language, such as a word or sentence, on the basis of the linguistic context. Second, we apply language models in an illustrative analysis to demonstrate the mechanics of using these models in social science research. The example application uses language models to classify names in a large administrative database; the classifications are then used to measure a sociologically important phenomenon: the spatial variation of religiosity. This application highlights several advantages of language models, including their effectiveness in classifying text that contains variation around the base structures, as is often the case with localized naming conventions and dialects. We conclude by discussing language models’ potential to contribute to sociological research beyond classification through their ability to generate language.
This article outlines a research agenda for a sociology of artificial intelligence (AI). The authors review two areas in which sociological theories and methods have made significant contributions to the study of inequalities and AI: (1) the politics of algorithms, data, and code and (2) the social shaping of AI in practice. The authors contrast sociological approaches that emphasize intersectional inequalities and social structure with other disciplines’ approaches to the social dimensions of AI, which often have a thin understanding of the social and emphasize individual-level interventions. This scoping article invites sociologists to use the discipline’s theoretical and methodological tools to analyze when and how inequalities are made more durable by AI systems. Sociologists have an ability to identify how inequalities are embedded in all aspects of society and to point toward avenues for structural social change. Therefore, sociologists should play a leading role in the imagining and shaping of AI futures.
The integration of social science with computer science and engineering fields has produced a new area of study: computational social science. This field applies computational methods to novel sources of digital data such as social media, administrative records, and historical archives to develop theories of human behavior. We review the evolution of this field within sociology via bibliometric analysis and in-depth analysis of the following subfields where this new work is appearing most rapidly: ( a) social network analysis and group formation; ( b) collective behavior and political sociology; ( c) the sociology of knowledge; ( d) cultural sociology, social psychology, and emotions; ( e) the production of culture; ( f ) economic sociology and organizations; and ( g) demography and population studies. Our review reveals that sociologists are not only at the center of cutting-edge research that addresses longstanding questions about human behavior but also developing new lines of inquiry about digital spaces as well. We conclude by discussing challenging new obstacles in the field, calling for increased attention to sociological theory, and identifying new areas where computational social science might be further integrated into mainstream sociology.
Expected final online publication date for the Annual Review of Sociology, Volume 46 is July 30, 2020. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
How do protest actions impact public support for social movements? Here we test the claim that extreme protest actions-protest behaviors perceived to be harmful to others, highly disruptive, or both-typically reduce support for social movements. Across 6 experiments, including 3 that were preregistered, participants indicated less support for social movements that used more extreme protest actions. This result obtained across a variety of movements (e.g., animal rights, anti-Trump, anti-abortion) and extreme protest actions (e.g., blocking highways, vandalizing property). Further, in 5 of 6 studies, negative reactions to extreme protest actions also led participants to support the movement's central cause less, and these effects were largely independent of individuals' prior ideology or views on the issue. In all studies we found effects were driven by diminished social identification with the movement. In Studies 4-6, serial mediation analyses detailed a more in-depth model: observers viewed extreme protest actions to be immoral, reducing observers' emotional connection to the movement and, in turn, reducing identification with and support for the movement. Taken together with prior research showing that extreme protest actions can be effective for applying pressure to institutions and raising awareness of movements, these findings suggest an activist's dilemma, in which the same protest actions that may offer certain benefits are also likely to undermine popular support for social movements. (PsycINFO Database Record (c) 2020 APA, all rights reserved).
The Fragile Families Challenge provided an opportunity to empirically assess the applicability of black-box machine learning models to sociological questions and the extent to which interpretable explanations can be extracted from these models. In this article the author uses neural network models to predict high school grade point average and examine how variations of basic network parameters affect predictive performance. Using a recently proposed technique, the author identifies the most important predictive variables used by the best performing model, finding that they relate to parenting and the child’s cognitive and behavioral development, consistent with prior work. The author concludes by discussing the implications of these findings for the relationship between prediction and explanation in sociological analyses.
Reproducibility is fundamental to science, and an important component of reproducibility is computational reproducibility: the ability of a researcher to recreate the results of a published study using the original author’s raw data and code. Although most people agree that computational reproducibility is important, it is still difficult to achieve in practice. In this article, the authors describe their approach to enabling computational reproducibility for the 12 articles in this special issue of Socius about the Fragile Families Challenge. The approach draws on two tools commonly used by professional software engineers but not widely used by academic researchers: software containers (e.g., Docker) and cloud computing (e.g., Amazon Web Services). These tools made it possible to standardize the computing environment around each submission, which will ease computational reproducibility both today and in the future. Drawing on their successes and struggles, the authors conclude with recommendations to researchers and journals.
Political discourse is the soul of democracy, but misunderstanding and conflict can fester in divisive conversations. The widespread shift to online discourse exacerbates many of these problems and corrodes the capacity of diverse societies to cooperate in solving social problems. Scholars and civil society groups promote interventions that make conversations less divisive or more productive, but scaling these efforts to online discourse is challenging. We conduct a large-scale experiment that demonstrates how online conversations about divisive topics can be improved with AI tools. Specifically, we employ a large language model to make real-time, evidence-based recommendations intended to improve participants’ perception of feeling understood. These interventions improve reported conversation quality, promote democratic reciprocity, and improve the tone, without systematically changing the content of the conversation or moving people’s policy attitudes.
This study analyzes large language models (LLMs) as a methodology for computational sociology, focusing on applications to supervised text classification. We consider how the latest generation of text-to-text transformer models can make predictions using prompts and minimal training examples and assess the sensitivity of these approaches to wording and composition. Through a comprehensive case study on identifying opinions expressed about politicians on Twitter and Facebook, we evaluate four different LLM architectures, varying in size, training data, and architecture. We compare the performance across different training regimes, from prompt-based zero-shot learning to fine-tuning using thousands of annotated examples. Our findings demonstrate how LLMs can perform complex text classification tasks with high accuracy, substantially outperforming conventional baselines. We use these results to provide practical recommendations for sociologists interested in employing LLMs for text classification tasks. Fine-tuning smaller models offers an optimal solution for most researchers due to their relatively high accuracy and low cost. We discuss the trade-offs between proprietary and open-source models, the importance of evaluating models for bias, and concerns related to transparency and reproducibility. This study contributes to understanding the capabilities and limitations of these models in a sociological context, providing a foundation for future research and applications in the field.
Many NLP applications require manual text annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using four samples of tweets and news articles (n = 6,183), we show that ChatGPT outperforms crowd workers for several annotation tasks, including relevance, stance, topics, and frame detection. Across the four datasets, the zero-shot accuracy of ChatGPT exceeds that of crowd workers by about 25 percentage points on average, while ChatGPT's intercoder agreement exceeds that of both crowd workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003-about thirty times cheaper than MTurk. These results demonstrate the potential of large language models to drastically increase the efficiency of text classification.
In this paper, I explore the transformative potential of Large Language Models (LLMs) such as ChatGPT in the realm of qualitative research, particularly in the social sciences. These generative AI models, trained on extensive textual data, have the unique ability to "understand," generate, and manipulate human-like text, offering unprecedented opportunities for data analysis and interpretation. I argue that LLMs, with this capacity, can significantly enhance the depth and efficiency of qualitative analysis. They can quickly identify patterns, themes, and sentiments in the data, providing a level of nuance that can be challenging to achieve with manual coding. Furthermore, their ability to generate human-like text can be used to simulate social interactions, create engaging presentations of research findings, and even "converse" with the data in a natural and flexible way. Indeed a central contribution of this paper lies in exploring this novel concept of "asking questions of" or "conversing with" text-based data, which opens up new avenues for qualitative research and analysis. This interactive capability of LLMs provides a transformative approach to topic coding and content analysis, allowing researchers to pose complex, nuanced questions to their data and receive responses in natural language. Ethical considerations and limitations are also discussed.
Visual data have dramatically increased in quantity in the digital age, presenting new opportunities for social science research. However, the extensive time and labor costs to process and analyze these data with existing approaches limit their use. Computer vision methods hold promise but often require large and nonexistent training data to identify sociologically relevant variables. We present a cost-efficient method for curating training data that utilizes simple tasks and pairwise comparisons to interpret and analyze visual data at scale using computer vision. We apply our approach to the detection of trash levels across space and over time in millions of street-level images in three physically distinct US cities. By comparing to ratings produced in a controlled setting and utilizing computational methods, we demonstrate generally high reliability in the method and identify sources that limit it. Altogether, this approach expands how visual data can be used at a large scale in sociology.
Transformer-based models for transfer learning have the potential to achieve high prediction accuracies on text-based supervised learning tasks with relatively few training data instances. These models are thus likely to benefit social scientists that seek to have as accurate as possible text-based measures, but only have limited resources for annotating training data. To enable social scientists to leverage these potential benefits for their research, this article explains how these methods work, why they might be advantageous, and what their limitations are. Additionally, three Transformer-based models for transfer learning, BERT, RoBERTa, and the Longformer, are compared to conventional machine learning algorithms on three applications. Across all evaluated tasks, textual styles, and training data set sizes, the conventional models are consistently outperformed by transfer learning with Transformers, thereby demonstrating the benefits these models can bring to text-based social science research.
The last decade witnessed a spectacular rise in the volume of available textual data. With this new abundance came the question of how to analyze it. In the social sciences, scholars mostly resorted to two well-established approaches, human annotation on sampled data on the one hand (either performed by the researcher, or outsourced to microworkers), and quantitative methods on the other. Each approach has its own merits - a potentially very fine-grained analysis for the former, a very scalable one for the latter - but the combination of these two properties has not yielded highly accurate results so far. Leveraging recent advances in sequential transfer learning, we demonstrate via an experiment that an expert can train a precise, efficient automatic classifier in a very limited amount of time. We also show that, under certain conditions, expert-trained models produce better annotations than humans themselves. We demonstrate these points using a classic research question in the sociology of journalism, the rise of a “horse race” coverage of politics. We conclude that recent advances in transfer learning help us augment ourselves when analyzing unstructured data.
Radical-right campaigns commonly employ three discursive elements: anti-elite populism, exclusionary and declinist nationalism, and authoritarianism. Recent scholarship has explored whether these frames have diffused from radical-right to centrist parties in the latter’s effort to compete for the former’s voters. This study instead investigates whether similar frames had been used by mainstream political actors prior to their exploitation by the radical right (in the U.S., Donald Trump’s 2016 and 2020 campaigns). To do so, we identify instances of populism, nationalism (i.e., exclusionary and inclusive definitions of national symbolic boundaries and displays of low and high national pride), and authoritarianism in the speeches of Democratic and Republican presidential nominees between 1952 and 2020. These frames are subtle, infrequent, and polysemic, which makes their measurement difficult. We overcome this by leveraging the affordances of neural language models—in particular, a robustly optimized variant of bidirectional encoder representations from Transformers (RoBERTa) and active learning. As we demonstrate, this approach is more effective for measuring discursive frames than other methods commonly used by social scientists. Our results suggest that what set Donald Trump’s campaign apart from those of mainstream presidential candidates was not the invention of a new form of politics, but the combination of negative evaluations of elites, low national pride, and authoritarianism—all of which had long been present among both parties—with an explicit evocation of exclusionary nationalism, which had been articulated only implicitly by prior presidential nominees. Radical-right discourse—at least at the presidential level in the United States—should therefore be characterized not as a break with the past but as an amplification and creative rearrangement of existing political-cultural tropes.
Public culture is a powerful source of cognitive socialization; for example, media language is full of meanings about body weight. Yet it remains unclear how individuals process meanings in public culture. We suggest that schema learning is a core mechanism by which public culture becomes personal culture. We propose that a burgeoning approach in computational text analysis – neural word embeddings – can be interpreted as a formal model for cultural learning. Embeddings allow us to empirically model schema learning and activation from natural language data. We illustrate our approach by extracting four lower-order schemas from news articles: the gender, moral, health, and class meanings of body weight. Using these lower-order schemas we quantify how words about body weight “fill in the blanks” about gender, morality, health, and class. Our findings reinforce ongoing concerns that machine-learning models (e.g., of natural language) can encode and reproduce harmful human biases.
As the field of computational text analysis within the social sciences is maturing, computational methods are no longer seen as ends in themselves, but rather as means toward answering theoretically motivated research questions. The objective of this special issue is to showcase such research: the use of novel computational methods in the service of advancing substantive scientific knowledge. In presenting the contributions to the issue, we discuss several insights that emerge from this work, which hold relevance not only for current and aspiring practitioners of computational text analysis, but also for its skeptics. These concern the central role of theory in designing and executing computational research, the selection of appropriate techniques from a rapidly growing methodological toolkit, the benefits—and risks—of methodological bricolage, and the necessity of validating all aspects of the research process. The result is a set of broad considerations concerning the effective application of computational methods to substantive questions, illustrated by eight exemplary empirical studies.
Research in organizational theory takes as a key premise the notion that organizations are “actors.” Organizational actorhood, or agency, depends, in part, on how external audiences perceive organizations. In other words, organizational agency requires that external audiences take organizations to be agents. Yet little empirical research has attempted to measure these attributions: when do audiences assume that organizations are agents and how have these attributions changed over time? In this article, I suggest that scholars can triangulate across computational methods—including named entity recognition, dependency parsing, topic models, and dictionary methods—to analyze attributions of agency in text, discourse that I term “agent talk.” I demonstrate the utility of this approach by analyzing how business organizations were discussed as agents during a key period of organizational development, the turn of the twentieth century. Analyzing articles from two of the leading national newspapers, the Wall Street Journal and New York Times, I examine agent talk in everyday business discourse. I find that agent talk generally increased over the early twentieth century, as organizations were depicted as active subjects in text and personified as speakers. Moreover, I find that this discourse was concentrated in social and legal semantic contexts: in particular, contexts relating to labor, regulation, and railroads. Finally, I show the uneven growth of this rhetoric over time, as organizations across different semantic arenas were personified as speakers. Overall, these results show how measures of discourse can provide a window into how and when audiences endow organizations with actorhood.
This paper considers the adoption of computational techniques within research designs modeled after the extended case method. Echoing calls to augment the power of contemporary researchers through the adoption of computational text analysis methods, we offer a framework for thinking about how such techniques can be integrated into quasi-ethnographic workflows to address broad, structural sociological claims. We focus, in particular, on how this adoption of novel forms of evidence impacts corpus design and interpretation (which we tie to matters of casing), theoretical elaboration (which we associate to moving empirical claims across scales and empirical domains), and verification (which we see as a process of reflexive scaffolding of theoretical claims). We provide an example of the use of this framework through a study of the marketization of social scientific knowledge in the United Kingdom.
Text as Data represents a major advance for teaching text analysis in the social sciences, digital humanities and data science by providing an integrated framework for how to conceptualize and deploy natural language processing techniques to enrich descriptive and causal analyses of social life in and from text. Here I review achievements of the book and highlight complementary paths not taken, including discussion of recent computational techniques like transformers, which have come to dominate automated language understanding and are just beginning to find their way into the careful research designs showcased in the book. These new methods not only highlight text as a signal from society, but textual models as simulations of society, which could fuel future advances in causal inference and experimentation. Text as Data's focus on textual discovery, measurement and inference points us toward this new frontier, cautioning us not to ignore, but build upon social scientific interpretation and theory.
The pairing of massive data sets with processes—or algorithms—written in computer code to sort through, organize, extract, or mine them has made inroads in almost every major social institution. This article proposes a reading of the scholarly literature concerned with the social implications of this transformation. First, we discuss the rise of a new occupational class, which we call the coding elite. This group has consolidated power through their technical control over the digital means of production and by extracting labor from a newly marginalized or unpaid workforce, the cybertariat. Second, we show that the implementation of techniques of mathematical optimization across domains as varied as education, medicine, credit and finance, and criminal justice has intensified the dominance of actuarial logics of decision-making, potentially transforming pathways to social reproduction and mobility but also generating a pushback by those so governed. Third, we explore how the same pervasive algorithmic intermediation in digital communication is transforming the way people interact, associate, and think. We conclude by cautioning against the wildest promises of artificial intelligence but acknowledging the increasingly tight coupling between algorithmic processes, social structures, and subjectivities.
Expected final online publication date for the Annual Review of Sociology, Volume 47 is July 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
We provide an introduction of the functioning, implementation, and challenges of convolutional neural networks (CNNs) to classify visual information in social sciences. This tool can help scholars to make more efficient the tedious task of classifying images and extracting information from them. We illustrate the implementation and impact of this methodology by coding handwritten information from vote tallies. Our paper not only demonstrates the contributions of CNNs to both scholars and policy practitioners, but also presents the practical challenges and limitations of the method, providing advice on how to deal with these issues.
The conventions ethnographers follow to gather, write about, and store their data are increasingly out of sync with contemporary research expectations and social life. Despite technological advancements that allow ethnographers to observe their subjects digitally and record interactions, few follow subjects online and many still reconstruct quotes from memory. Amid calls for data transparency, ethnographers continue to conceal subjects’ identities and keep fieldnotes private. But things are changing. We review debates, dilemmas, and innovations in ethnography that have arisen over the past two decades in response to new technologies and calls for transparency. We focus on emerging conversations around how ethnographers record, collect, anonymize, verify, and share data. Considering the replication crisis in the social sciences, we ask how ethnographers can enable others to reanalyze their findings. We address ethical implications and offer suggestions for how ethnographers can develop standards for transparency that are consistent with their commitment to their subjects and interpretive scholarship.
Expected final online publication date for the Annual Review of Sociology, Volume 47 is July 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Protest event analysis is an important method for the study of collective action and social movements and typically draws on traditional media reports as the data source. We introduce collective action from social media (CASM)—a system that uses convolutional neural networks on image data and recurrent neural networks with long short-term memory on text data in a two-stage classifier to identify social media posts about offline collective action. We implement CASM on Chinese social media data and identify more than 100,000 collective action events from 2010 to 2017 (CASM-China). We evaluate the performance of CASM through cross-validation, out-of-sample validation, and comparisons with other protest data sets. We assess the effect of online censorship and find it does not substantially limit our identification of events. Compared to other protest data sets, CASM-China identifies relatively more rural, land-related protests and relatively few collective action events related to ethnic and religious conflict.
Black box machine learning models are currently being used for high-stakes decision making throughout society, causing problems in healthcare, criminal justice and other domains. Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practice and can potentially cause great harm to society. The way forward is to design models that are inherently interpretable. This Perspective clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare and computer vision. There has been a recent rise of interest in developing methods for ‘explainable AI’, where models are created to explain how a first ‘black box’ machine learning model arrives at a specific decision. It can be argued that instead efforts should be directed at building inherently interpretable models in the first place, in particular where they are applied in applications that directly affect human lives, such as in healthcare and criminal justice.
Machine learning is a field at the intersection of statistics and computer science that uses algorithms to extract information and knowledge from data. Its applications increasingly find their way into economics, political science, and sociology. We offer a brief introduction to this vast toolbox and illustrate its current uses in the social sciences, including distilling measures from new data sources, such as text and images; characterizing population heterogeneity; improving causal inference; and offering predictions to aid policy decisions and theory development. We argue that, in addition to serving similar purposes in sociology, machine learning tools can speak to long-standing questions on the limitations of the linear modeling framework, the criteria for evaluating empirical findings, transparency around the context of discovery, and the epistemological core of the discipline.
Expected final online publication date for the Annual Review Sociology Volume 45 is July 30, 2019. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.