John A. Bateman

John A. Bateman
Universität Bremen | Uni Bremen · English-Speaking Cultures, Linguistics, Transmedial Textuality, Spatial Cognition

PhD (Artificial Intelligence)

About

329
Publications
93,415
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
5,961
Citations
Citations since 2017
58 Research Items
2792 Citations
20172018201920202021202220230200400600
20172018201920202021202220230200400600
20172018201920202021202220230200400600
20172018201920202021202220230200400600
Introduction
John Bateman is Professor of Appliable Linguistics in the English and Linguistics Departments of Bremen University. His research areas include functional linguistic approaches to multimodal document design, the semiotics of film and other media, multimodal semiotics, computational dialogue systems, formal ontology, and discourse semantics. He has been investigating the relation between language and other semiotic systems for many years, publishing widely in all these areas. ... [N.B. I DO NOT READ MESSAGES ON THIS PLATFORM, NEITHER DO I PROVIDE COPIES OF BOOKS; IF YOU WISH TO CONTACT ME, PLEASE DO SO WITH REGULAR EMAIL. ]
Additional affiliations
April 1999 - present
Universität Bremen
Position
  • Prof. Linguistics
Description
  • http://www-user.uni-bremen.de/~bateman
September 1997 - March 1999
University of Stirling
Position
  • Lecturer

Publications

Publications (329)
Chapter
Full-text available
As the use and diversity of diagrams across many disciplines grows, there is an increasing interest in the diagrams research community concerning how such diversity might be documented and explained. In this article, we argue that one way of achieving increased reliability, coverage, and utility for a general classification of diagrams is to draw o...
Article
Full-text available
Going from natural language directions to fully specified executable plans for household robots involves a challenging variety of reasoning steps. In this paper, a processing pipeline to tackle these steps for natural language directions is proposed and implemented. It uses the ontological Socio-physical Model of Activities (SOMA) as a common inter...
Article
In this position statement, I draw broadly on approaches and theories relevant to the phenomena of multimodality to propose some methodological directions for further cycles of development and application. Two apparently opposed goals are accepted and reconciled: on the one hand, increased attention needs to be paid to empirical studies and, on the...
Article
Full-text available
In this article, we argue for the benefits of combining large-scale analyses of visual materials currently pursued within digital humanities with insights from multimodality research, which is an emerging discipline that studies how human communication relies on appropriate combinations of expressive resources. We show that concepts developed withi...
Chapter
Full-text available
In this paper, we present foundations of the Socio-physical Model of Activities (SOMA). SOMA represents both the physical as well as the social context of everyday activities. Such tasks seem to be trivial for humans, however, they pose severe problems for artificial agents. For starters, a natural language command requesting something will leave m...
Article
Full-text available
This article presents results of an exploratory investigation combining multimodal cohesion analysis and eye-tracking studies. Multimodal cohesion, as a tool of multimodal discourse analysis, goes beyond linguistic cohesive mechanisms to enable the construction of cross-modal discourse structures that systematically relate technical details of audi...
Article
The broad field of ‘multimodality’ covers a rather diverse collection of approaches and perspectives whose greatest common factor is that they investigate communicative situations where distinct forms of expression appear to be synergistically combined. The precise definition of what constitutes a distinct form of expression varies across schools o...
Article
GUM is a linguistically-motivated ontology originally developed to support natural language processing systems by offering a level of representation intermediate between linguistic forms and domain knowledge. Whereas modeling decisions for individual domains may need to be responsive to domain-specific criteria, a linguistically-motivated ontology...
Chapter
Multimodality research has always shown a strong reliance on data. However , the field has primarily developed around more exploratory, descriptive, and interpretative work on smaller data sets-as suggested by results we present from a meta-study of contributions to three multimodality-close international journals (Social Semiotics, Visual Communic...
Book
This volume advances the data-based study of multimodal artefacts and performances by showcasing methods and results from the latest endeavors in empirical multimodal research, representing a vibrant international and interdisciplinary research community. The collated chapters identify and seek to inspire novel, mixed-method approaches to investiga...
Article
Many studies investigating the use and effectiveness of multimodal communication are now confronting the need to engage with larger bodies of data in order to achieve more empirically robust accounts, moving beyond the earlier prevalence of small-scale ‘case studies’. In this article, I briefly characterise how recent developments in the theory of...
Article
Full-text available
This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural sciences, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowdsourced...
Article
This essay addresses the nature of so–called ‘digital media’ in a literacy context from the perspectives of semiotics, theories of the ‘medium’, and computation. It argues that most accounts that attempt to work with some notion of ‘digital media’ anchor themselves insufficiently in semiotics and computation and the essential combination of these t...
Article
Full-text available
Explanation videos are increasingly common on media websites such as YouTube and are used by school students, university students, and members of the general public alike. Such videos cover all areas of knowledge and aim to provide viewer-appropriate explanations concerning a large variety of topics. It is, however, still far from clear how such vi...
Conference Paper
Full-text available
In this paper, we present foundations of the Socio-physical Model of Activities (SOMA). SOMA represents both the physical as well as the social context of everyday activities. Such tasks seem to be trivial for humans, however, they pose severe problems for artificial agents. For starters, a natural language command requesting something will leave m...
Preprint
Full-text available
In this article, we bring together theories of multimodal communication and computational methods to study how primary school science diagrams combine multiple expressive resources. We position our work within the field of digital humanities, and show how annotations informed by multimodality research, which target expressive resources and discours...
Preprint
Full-text available
In this paper, we present foundations of the Socio-physical Model of Activities (SOMA). SOMA represents both the physical as well as the social context of everyday activities. Such tasks seem to be trivial for humans, however, they pose severe problems for artificial agents. For starters, a natural language command requesting something will leave m...
Chapter
Functional relations such as containment or support have proven difficult to formalize. Although previous efforts have attempted this using hybrids of several theories, from mereology to temporal logic, we find that such purely symbolic approaches do not account for the embodied nature of functional relations, i.e. that they are used by embodied ag...
Article
Full-text available
One of the problems that service robotics deals with is to bring mobile manipulators to work in semi-structured human scenarios, which requires an efficient and flexible way to execute everyday tasks, like serve a cup in a cluttered environment. Usually, for those tasks, the combination of symbolic and geometric levels of planning is necessary, as...
Conference Paper
Full-text available
One of the key reasoning tasks of robotic agents is inferring possible actions that can be accomplished with a given object at hand. This cognitive task is commonly referred to as inferring the affordances of objects. In this paper, we propose a novel conceptualization of affordances and its realization as a description logic ontology. The key idea...
Preprint
Full-text available
In this article, we propose a multimodal perspective to diagrammatic representations by sketching a description of what may be tentatively termed the diagrammatic mode. We consider diagrammatic representations in the light of contemporary multimodality theory and explicate what enables diagrammatic representations to integrate natural language, var...
Preprint
Full-text available
Although the number of articles in visual and multimodal communication that include statistical validation of claimed results is increasing, we suggest in this article that this is by no means enough. Statistical methods should belong to every multimodality researcher's toolset precisely because the phenomena under study are subtle and complex. Wit...
Article
Despite a long association between information design and semiotics, connections remain limited in many respects. This contribution argues that one reason for this is the traditionally weak connection between semiotics and empirical methods. To counter this, a model of multimodal communication is introduced in which theoretical description and empi...
Preprint
Full-text available
This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural science, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowd-sourced...
Chapter
This afterword draws insights and conclusions from the preceding chapters by critically engaging once again with the disciplinary status of multimodality. It explicates the main points of discussion of the contributions and makes some recommendations concerning disciplinarity and multimodality. The path taken addresses multimodality at a fundamenta...
Chapter
In this introduction, we discuss the idea of establishing a discipline of multimodality, considering both how this might be defined and potential benefits and challenges of attaining such an independent status. This builds on previous rounds of discussion within the Bremen Conferences on Multimodality (BreMM) series concerning this issue, where div...
Chapter
Full-text available
In this paper, we introduce the framework of MEANinGS for the semi-autonomous accumulation of world knowledge for robots. Where manual aggregation is inefficient and prone to incompleteness and autonomous approaches suffer from underspecified information, we deploy the human computation game Kitchen Clash and give evidence of its efficiency, comple...
Book
Multimodality’s popularity as a semiotic approach has not resulted in a common voice yet. Its conceptual anchoring as well as its empirical applications often remain localized and disparate, and ideas of a theory of multimodality are heterogeneous and uncoordinated. For the field to move ahead, it must achieve a more mature status of reflection, mu...
Chapter
Full-text available
Autonomous indoor robots are supposed to accomplish tasks, like serve a cup, which involves manipulation actions, where task and motion planning levels are coupled. In both planning levels and execution phases, several sources of failures can occur. In this paper, an interpretation ontology covering several sources of failures in automated planning...
Article
The phenomena of mixing, blending, and referencing media is a major topic in contemporary media studies. Finding a sufficient semiotic foundation to characterize such phenomena remains challenging. The current article argues that combining a notion of ‘semiotic mode' developed within the field of multimodality with a Peircean foundation contributes...
Article
Page layout is one of the most salient features of graphic novels and comics that readers encounter: even before engaging with specific content, an overall impression of the page composition will have already been communicated. In the critical literature on comics and graphic novels, it is also commonly claimed that page composition plays a signifi...
Article
The Cambridge Handbook of Systemic Functional Linguistics - edited by Geoff Thompson May 2019
Chapter
Störungen der Genreerwartung entstehen, wenn etablierte Muster durch narrative, dramaturgische, allgemein ästhetische oder andere Strategien unterlaufen werden. Sie realisieren sich in einer inhaltlich unmittelbaren Darstellung des Gesagten, dem Was des Filmtextes (histoire), aber auch im formalen Modus, im Wie des Gegebenen (discours). Stören und...
Chapter
Recurrent neural networks have found applications in NLP, but their operation is difficult to interpret. A state automaton that approximates the network would be more interpretable, but for this one needs a method to group network activation states by their behavior. In this paper we propose such a method, and compare it to an existing dimensionali...
Article
Ledin and Machin's critique of the use of some current approaches to multimodality for the purposes of critical discourse studies raises some important methodological concerns that need to be addressed. However, both the particular position they develop as well as some of the key points they raise are themselves problematic. In this response, I arg...
Conference Paper
As robots are expected to accomplish human-level manipulation tasks, the demand for formal knowledge representation techniques and reasoning for robots increases dramatically. In this paper we describe how to make use of heterogeneous ontologies in service robotics. To illustrate the vision, we take the action of pouring as an example.
Article
Full-text available
Educational content of many kinds and from many disciplines are increasingly presented in the form of short videos made broadly accessible via platforms such as YouTube. We argue that understanding how such communicative forms function effectively (or not) demands a more thorough theoretical foundation in the principles of multimodal communication...
Article
Full-text available
The effective study of transmedia adaptation requires descriptions that allow us to track how changes in media may correlate with both similarities and differences across medial realisations of a work. To the extent that such description can be made systematic and reliable, it becomes possible to apply a variety of empirical methods for revealing r...
Article
Full-text available
The article deals with fundamental and broadly discussed issues within the paradigm of multimodal analysis concerning how the intersemiotic interplay of semiotic resources constructs meaning. Taking a specifically linguistic focus on theories and methods within this paradigm, the article orientates particularly towards discourse analytical and text...
Article
This short position paper argues that new semiotically-anchored approaches to multimodality offer much for other disciplines now engaging with multimodality. In particular, the account of multimodality introduced is argued to position current discussions of the potential role of multimodality in argumentation studies more effectively, untangling se...
Article
The account of signs, signification and meaning set out by the philosopher Charles Sanders Peirce around the beginning of the twentieth century is a foundation stone of modern semiotics. In Peirce’s conception, semiotics concerned the process of signification at its most general and was intrinsically multimodal. It is then logical that contemporary...
Article
A prerequisite for approaching the study of changes across media and their evolving roles in society, especially when ‘new’ media emerge, is that one has a good theoretical grasp of just what ‘media’ are and how they may be approached analytically. To support insightful analysis going beyond description and cataloguing, there is a need to make curr...
Book
Full-text available
This textbook provides the first foundational introduction to the practice of analysing multimodality, covering the full breadth of media and situations in which multimodality needs to be a concern. Readers learn via use cases how to approach any multimodal situation and to derive their own specifically tailored sets of methods for conducting and e...
Article
This article presents a mixed methods approach for analysing text and image relations in violent extremist discourse. The approach involves integrating multimodal discourse analysis with data mining and information visualisation, resulting in theoretically informed empirical techniques for automated analysis of text and image relations in large dat...
Book
Full-text available
This book examines film as a multimodal text and an audiovisual synthesis, bringing together current work within the fields of narratology, philosophy, multimodal analysis, sound as well as cultural studies in order to cover a wide range of international academic interest. The book provides new insights into current work and turns the discussion to...
Article
Analytic interest in comics, graphic novels and similarly visual media is currently experiencing considerable growth. In order to pursue empirical investigation of such media, it is useful to explore how data of this kind can be made accessible for the application of established empirical methods, such as linguistic corpus analysis. Many forms of c...
Article
This article demonstrates how a digital environment offers new opportunities for transforming qualitative data into quantitative data in order to use data mining and information visualization for mixed methods research. The digital approach to mixed methods research is illustrated by a framework which combines qualitative methods of multimodal disc...
Article
The digital turn in visual studies has played a major role in the terminological overlap between ‘archive’, ‘database’ and ‘corpus’, and it has brought about a number of positive developments such as improved accessibility and availability. At the same time, it has also raised important questions pertaining to the materiality, searchability, annota...
Chapter
In this article, we summarize and review approaches to the analysis of comics within a German research perspective that have adopted a predominantly linguistic or textlinguistic orientation. We focus particularly on research combining both linguistics and semiotics in order to treat verbal and graphical material as integral components of unified ac...
Book
Full-text available
Semiotics has been making progressively inroads into marketing research over the past thirty years. Despite the amply demonstrated conceptual appeal and empirical pertinence of semiotic perspectives in various marketing research streams, spanning consumer research, brand communications, branding and consumer cultural studies, there has been a marke...
Conference Paper
Full-text available
In this position paper, we argue that advances in intelligent cinematography require better models of the multimodal structure of filmic discourse, and of the inferences made by an audience while films are being watched. Such questions have been addressed by film scholars and cognitive scientists in the past, but their models have not so far had su...
Article
This contribution discusses the potential role that could be played by dynamic discourse semantics as developed within formal and functional linguistic approaches to connected discourse for revitalising general semiotic approaches to complex multimodal artefacts and performances. By adopting such dynamic semantics as an integral part of a new defin...
Chapter
Full-text available
Conceptual blending has been employed very successfully to understand the process of concept invention, studied particularly within cognitive psychology and linguistics. However, despite this influential research, within computational creativity little effort has been devoted to fully formalise these ideas and to make them amenable to computational...
Article
Full-text available
The paper deals with the question of filling in the gutter or gap between comic panels by inferences and abductive reasoning. This is a highly discussed topic in comic studies and we choose a new and innovative way of approaching it, namely by combining linguistic, multimodal discourse analytical as well as logical and formal accounts in order to d...
Article
There have been many attempts to provide accounts of visually expressed narratives by drawing on our understandings of linguistic discourse. Such approaches have however generally proceeded piecemeal --- particular phenomena appearing similar to phenomena in verbal discourse are selected for discussion with insufficient consideration of just what i...
Article
Although regularly contested and rejected, the idea of a ‘grammar of comics’ continues to be applied to the analysis of comics in many disciplines and frameworks. The motivation for this is the evident regularity and systematicity exhibited by the comics medium as a form of expressive communication. Less commonly realized is that nowadays there are...
Chapter
Die nachfolgend skizzierte Versuchsanordnung zur Blickerfassung und ihre Ergeb-nisse stehen in einem größeren Forschungszusammenhang, der sich der Generie-rung soziokultureller Muster über audiovisuelle Medien widmet, insbesondere aus der Rezeption des Kinospielfilms. Es ist hier nicht der Ort, das ausführlich zu refe-rieren, aber es scheint den Au...
Article
The treatment of space and spatial language has long been an area of philosophical, theoretical, and now also increasingly practical concern. Detailed studies of spatial language and its use have been pursued in the areas of language typology, formal semantics, psycholinguistics, child language development, language relativity, human-machine intera...
Book
Text and image are used together in an increasingly flexible fashion and many disciplines and areas of study are now attempting to understand how these combinations work. This introductory textbook explores and analyses the various approaches to multimodality and offers a broad, interdisciplinary survey of all aspects of the text-image relation. It...
Article
Full-text available
In this paper we show that some notions from the textual organisation of verbal texts appear also to give insights to the organisation of films. In particular, the beginnings of films are suggested to operate as indicators of those films’ ‘method of development’ and so serve to set up expectations for guiding hypotheses and selective attention duri...
Article
Full-text available
The 9/11 attacks in the USA had profound political consequences at both domestic and international levels. Specific and controversial policy developments were pursued requiring substantial legitimation to find acceptance. A prime example was the USA PATRIOT Act, which was passed in the immediate aftermath of 9/11 and subsequently received considera...
Article
Ernest Davis' article “Qualititative Spatial Reasoning in Interpreting Text and Narrative” discusses challenges that the interpretation of natural language appears to raise for the formalization of commonsense spatial reasoning. Davis finds these to be of “surprising logical complexity,” but also “erratic” in that they do not show a logical structu...
Article
Full-text available
http://dx.doi.org/10.5007/2175-8026.2013n64p49 In this paper, the predominantly visual framework developed for the analysis of static multimodal documents within the Genre and Multimodality project ‘GeM’ is considered as a foundation for treating non-static multimodal artifacts. The paper introduces the original framework and characterizes how it...
Article
Full-text available
In this article, we apply methods under development in socio-functional semiotics to explore the transfer of resources originally developed for comics to the medium of film. We illustrate this concretely with respect to extracts taken from Ang Lee's Hulk (2003), a film we consider unique in its particular appropriation of expressive resources from...
Article
Although linguistically inflected semiotic approaches to film were pursued in the 1960s and 1970s, they have since been almost universally rejected within film studies and film theory even though film is precisely the kind of intentionally produced communicative artifact for which one would have expected semiotics to have much to offer. Traditional...
Article
This contribution discusses the potential role that could be played by dynamic discourse semantics as developed within formal and functional linguistic approaches to connected discourse for revitalising general semiotic approaches to complex multimodal artefacts and performances. By adopting such dynamic semantics as an integral part of a new defin...
Chapter
Mit diesem ersten Buch der Reihe, Film, Text, Kultur. Beiträge zur Textualität des Films wird das Spektrum aktueller Forschungsansätze aus verschiedenen Teildisziplinen ausgeleuchtet; zugleich liefert der Band neue Impulse zu einer zeitgemäßen interdisziplinären Film- und Textualitätsforschung.
Chapter
Film is an obvious candidate for consideration as a multimodal artifact.
Chapter
Nowadays there is a considerable awareness of the necessity of studying language and language use in association with accompanying modes of communication.Keywords:21st century;discourse analysis;methods;research methods in applied linguistics;computational methods and data processing;corpus