John A. Bateman

University of Bremen | Uni Bremen · English-Speaking Cultures, Linguistics, Transmedial Textuality, Spatial Cognition

A challenge for automated user access to, or presentation of, event data is the fact that such data rarely explicitly controls for appropriate narrativisation choices. This is problematic because readers nevertheless read narrativisation effects into texts regardless of whether those effects were intended or not. Gaining control of such effects, wh...

The pitch is a central part of accelerator programs commonly presented with a slide presentation, called pitch deck. This study seeks to understand the ways in which pitch decks are structured. A slide-based approach was taken to describe the structure of 96 pitch decks created at the Start-up Chile accelerator program. Results showed that 7 topics...

In this paper, we consider the issue of how the fine-grained multimodal design of educational explanation videos, such as those widely available on YouTube and other platforms, may be made accessible to empirical studies of reception and effectiveness. This is necessary because previous research has often led to conflicting conclusions concerning t...

We present the Meta-Ontology for Introspection (MOI): Inspired by fundamental processes of the human mind, cognitive architectures (CAs) explore ever more methods to leverage metacognition. Still, an ontological model to trace metacognitive experiences for learning or as input for metacognitive control routines has yet to be developed. Based on a r...

News reporting has long been seen as involving a form of storytelling but techniques for revealing narrative constructions in audiovisual news remain limited. As the forms of expression mobilised for news become ever more diverse and multimodal, the challenges posed for analysis grow accordingly. The present paper asks to what extent we can pursue...

The analysis of news dissemination is of utmost importance since the credibility of information and the identification of disinformation and misinformation affect society as a whole. Given the large amounts of news data published daily on the Web, the empirical analysis of news with regard to research questions and the detection of problematic news...

Mit dem begrifflichen Paradox der bestimmten Unbestimmtheit wird der Zwiespalt umrissen zwischen der offenen Struktur von filmischen (und anderen audiovisuellen) Artefakten einerseits und der funktionalen Lenkung in ihnen andererseits. Es geht darum, auf sehr unterschiedlichen Ebenen Markierungen zu identifizieren, die bestimmte kognitive und emoti...

This paper treats dance as a movement-based semiotic system, focusing on classical ballet as an example in order to show how dance can be made accessible to both detailed description and empirical investigation as a form of communication. The study contributes to a growing tradition of multidisciplinary research that looks at a variety of dance for...

Review of: Who Understands Comics? Questioning the Universality of Visual Language Comprehension , Neil Cohn (2020) London and New York: Bloomsbury Academic, 256 pp., ISBN 978-1-35015-604-3, p/bk, £25.99

As the use and diversity of diagrams across many disciplines grows, there is an increasing interest in the diagrams research community concerning how such diversity might be documented and explained. In this article, we argue that one way of achieving increased reliability, coverage, and utility for a general classification of diagrams is to draw o...

Going from natural language directions to fully specified executable plans for household robots involves a challenging variety of reasoning steps. In this paper, a processing pipeline to tackle these steps for natural language directions is proposed and implemented. It uses the ontological Socio-physical Model of Activities (SOMA) as a common inter...

In this position statement, I draw broadly on approaches and theories relevant to the phenomena of multimodality to propose some methodological directions for further cycles of development and application. Two apparently opposed goals are accepted and reconciled: on the one hand, increased attention needs to be paid to empirical studies and, on the...

In this article, we argue for the benefits of combining large-scale analyses of visual materials currently pursued within digital humanities with insights from multimodality research, which is an emerging discipline that studies how human communication relies on appropriate combinations of expressive resources. We show that concepts developed withi...

In this paper, we present foundations of the Socio-physical Model of Activities (SOMA). SOMA represents both the physical as well as the social context of everyday activities. Such tasks seem to be trivial for humans, however, they pose severe problems for artificial agents. For starters, a natural language command requesting something will leave m...

This article presents results of an exploratory investigation combining multimodal cohesion analysis and eye-tracking studies. Multimodal cohesion, as a tool of multimodal discourse analysis, goes beyond linguistic cohesive mechanisms to enable the construction of cross-modal discourse structures that systematically relate technical details of audi...

The broad field of ‘multimodality’ covers a rather diverse collection of approaches and perspectives whose greatest common factor is that they investigate communicative situations where distinct forms of expression appear to be synergistically combined. The precise definition of what constitutes a distinct form of expression varies across schools o...

GUM is a linguistically-motivated ontology originally developed to support natural language processing systems by offering a level of representation intermediate between linguistic forms and domain knowledge. Whereas modeling decisions for individual domains may need to be responsive to domain-specific criteria, a linguistically-motivated ontology...

Multimodality research has always shown a strong reliance on data. However , the field has primarily developed around more exploratory, descriptive, and interpretative work on smaller data sets-as suggested by results we present from a meta-study of contributions to three multimodality-close international journals (Social Semiotics, Visual Communic...

This volume advances the data-based study of multimodal artefacts and performances by showcasing methods and results from the latest endeavors in empirical multimodal research, representing a vibrant international and interdisciplinary research community. The collated chapters identify and seek to inspire novel, mixed-method approaches to investiga...

Many studies investigating the use and effectiveness of multimodal communication are now confronting the need to engage with larger bodies of data in order to achieve more empirically robust accounts, moving beyond the earlier prevalence of small-scale ‘case studies’. In this article, I briefly characterise how recent developments in the theory of...

This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural sciences, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowdsourced...

This essay addresses the nature of so–called ‘digital media’ in a literacy context from the perspectives of semiotics, theories of the ‘medium’, and computation. It argues that most accounts that attempt to work with some notion of ‘digital media’ anchor themselves insufficiently in semiotics and computation and the essential combination of these t...

Explanation videos are increasingly common on media websites such as YouTube and are used by school students, university students, and members of the general public alike. Such videos cover all areas of knowledge and aim to provide viewer-appropriate explanations concerning a large variety of topics. It is, however, still far from clear how such vi...

In this paper, we present foundations of the Socio-physical Model of Activities (SOMA). SOMA represents both the physical as well as the social context of everyday activities. Such tasks seem to be trivial for humans, however, they pose severe problems for artificial agents. For starters, a natural language command requesting something will leave m...

In this article, we bring together theories of multimodal communication and computational methods to study how primary school science diagrams combine multiple expressive resources. We position our work within the field of digital humanities, and show how annotations informed by multimodality research, which target expressive resources and discours...

In this paper, we present foundations of the Socio-physical Model of Activities (SOMA). SOMA represents both the physical as well as the social context of everyday activities. Such tasks seem to be trivial for humans, however, they pose severe problems for artificial agents. For starters, a natural language command requesting something will leave m...

Functional relations such as containment or support have proven difficult to formalize. Although previous efforts have attempted this using hybrids of several theories, from mereology to temporal logic, we find that such purely symbolic approaches do not account for the embodied nature of functional relations, i.e. that they are used by embodied ag...

One of the problems that service robotics deals with is to bring mobile manipulators to work in semi-structured human scenarios, which requires an efficient and flexible way to execute everyday tasks, like serve a cup in a cluttered environment. Usually, for those tasks, the combination of symbolic and geometric levels of planning is necessary, as...

One of the key reasoning tasks of robotic agents is inferring possible actions that can be accomplished with a given object at hand. This cognitive task is commonly referred to as inferring the affordances of objects. In this paper, we propose a novel conceptualization of affordances and its realization as a description logic ontology. The key idea...

In this article, we propose a multimodal perspective to diagrammatic representations by sketching a description of what may be tentatively termed the diagrammatic mode. We consider diagrammatic representations in the light of contemporary multimodality theory and explicate what enables diagrammatic representations to integrate natural language, var...

Although the number of articles in visual and multimodal communication that include statistical validation of claimed results is increasing, we suggest in this article that this is by no means enough. Statistical methods should belong to every multimodality researcher's toolset precisely because the phenomena under study are subtle and complex. Wit...

Despite a long association between information design and semiotics, connections remain limited in many respects. This contribution argues that one reason for this is the traditionally weak connection between semiotics and empirical methods. To counter this, a model of multimodal communication is introduced in which theoretical description and empi...

This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural science, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowd-sourced...

This afterword draws insights and conclusions from the preceding chapters by critically engaging once again with the disciplinary status of multimodality. It explicates the main points of discussion of the contributions and makes some recommendations concerning disciplinarity and multimodality. The path taken addresses multimodality at a fundamenta...

In this introduction, we discuss the idea of establishing a discipline of multimodality, considering both how this might be defined and potential benefits and challenges of attaining such an independent status. This builds on previous rounds of discussion within the Bremen Conferences on Multimodality (BreMM) series concerning this issue, where div...

In this paper, we introduce the framework of MEANinGS for the semi-autonomous accumulation of world knowledge for robots. Where manual aggregation is inefficient and prone to incompleteness and autonomous approaches suffer from underspecified information, we deploy the human computation game Kitchen Clash and give evidence of its efficiency, comple...

Multimodality’s popularity as a semiotic approach has not resulted in a common voice yet. Its conceptual anchoring as well as its empirical applications often remain localized and disparate, and ideas of a theory of multimodality are heterogeneous and uncoordinated. For the field to move ahead, it must achieve a more mature status of reflection, mu...

Autonomous indoor robots are supposed to accomplish tasks, like serve a cup, which involves manipulation actions, where task and motion planning levels are coupled. In both planning levels and execution phases, several sources of failures can occur. In this paper, an interpretation ontology covering several sources of failures in automated planning...

The phenomena of mixing, blending, and referencing media is a major topic in contemporary media studies. Finding a sufficient semiotic foundation to characterize such phenomena remains challenging. The current article argues that combining a notion of ‘semiotic mode' developed within the field of multimodality with a Peircean foundation contributes...

Page layout is one of the most salient features of graphic novels and comics that readers encounter: even before engaging with specific content, an overall impression of the page composition will have already been communicated. In the critical literature on comics and graphic novels, it is also commonly claimed that page composition plays a signifi...

The Cambridge Handbook of Systemic Functional Linguistics - edited by Geoff Thompson May 2019

Störungen der Genreerwartung entstehen, wenn etablierte Muster durch narrative, dramaturgische, allgemein ästhetische oder andere Strategien unterlaufen werden. Sie realisieren sich in einer inhaltlich unmittelbaren Darstellung des Gesagten, dem Was des Filmtextes (histoire), aber auch im formalen Modus, im Wie des Gegebenen (discours). Stören und...

Recurrent neural networks have found applications in NLP, but their operation is difficult to interpret. A state automaton that approximates the network would be more interpretable, but for this one needs a method to group network activation states by their behavior. In this paper we propose such a method, and compare it to an existing dimensionali...

Ledin and Machin's critique of the use of some current approaches to multimodality for the purposes of critical discourse studies raises some important methodological concerns that need to be addressed. However, both the particular position they develop as well as some of the key points they raise are themselves problematic. In this response, I arg...

As robots are expected to accomplish human-level manipulation tasks, the demand for formal knowledge representation techniques and reasoning for robots increases dramatically. In this paper we describe how to make use of heterogeneous ontologies in service robotics. To illustrate the vision, we take the action of pouring as an example.

Educational content of many kinds and from many disciplines are increasingly presented in the form of short videos made broadly accessible via platforms such as YouTube. We argue that understanding how such communicative forms function effectively (or not) demands a more thorough theoretical foundation in the principles of multimodal communication...

Assembly recipes can elegantly be represented in description logic theories. With such a recipe, the robot can figure out the next assembly step through logical inference. However, before performing an action, the robot needs to ensure various spatial constraints are met, such as that the parts to be put together are reachable, non occluded, etc. S...

The effective study of transmedia adaptation requires descriptions that allow us to track how changes in media may correlate with both similarities and differences across medial realisations of a work. To the extent that such description can be made systematic and reliable, it becomes possible to apply a variety of empirical methods for revealing r...

The article deals with fundamental and broadly discussed issues within the paradigm of multimodal analysis concerning how the intersemiotic interplay of semiotic resources constructs meaning. Taking a specifically linguistic focus on theories and methods within this paradigm, the article orientates particularly towards discourse analytical and text...

This short position paper argues that new semiotically-anchored approaches to multimodality offer much for other disciplines now engaging with multimodality. In particular, the account of multimodality introduced is argued to position current discussions of the potential role of multimodality in argumentation studies more effectively, untangling se...

The account of signs, signification and meaning set out by the philosopher Charles Sanders Peirce around the beginning of the twentieth century is a foundation stone of modern semiotics. In Peirce’s conception, semiotics concerned the process of signification at its most general and was intrinsically multimodal. It is then logical that contemporary...

A prerequisite for approaching the study of changes across media and their evolving roles in society, especially when ‘new’ media emerge, is that one has a good theoretical grasp of just what ‘media’ are and how they may be approached analytically. To support insightful analysis going beyond description and cataloguing, there is a need to make curr...

This textbook provides the first foundational introduction to the practice of analysing multimodality, covering the full breadth of media and situations in which multimodality needs to be a concern. Readers learn via use cases how to approach any multimodal situation and to derive their own specifically tailored sets of methods for conducting and e...

This article presents a mixed methods approach for analysing text and image relations in violent extremist discourse. The approach involves integrating multimodal discourse analysis with data mining and information visualisation, resulting in theoretically informed empirical techniques for automated analysis of text and image relations in large dat...

This book examines film as a multimodal text and an audiovisual synthesis, bringing together current work within the fields of narratology, philosophy, multimodal analysis, sound as well as cultural studies in order to cover a wide range of international academic interest. The book provides new insights into current work and turns the discussion to...

Analytic interest in comics, graphic novels and similarly visual media is currently experiencing considerable growth. In order to pursue empirical investigation of such media, it is useful to explore how data of this kind can be made accessible for the application of established empirical methods, such as linguistic corpus analysis. Many forms of c...

This article demonstrates how a digital environment offers new opportunities for transforming qualitative data into quantitative data in order to use data mining and information visualization for mixed methods research. The digital approach to mixed methods research is illustrated by a framework which combines qualitative methods of multimodal disc...

The digital turn in visual studies has played a major role in the terminological overlap between ‘archive’, ‘database’ and ‘corpus’, and it has brought about a number of positive developments such as improved accessibility and availability. At the same time, it has also raised important questions pertaining to the materiality, searchability, annota...

In this article, we summarize and review approaches to the analysis of comics within a German research perspective that have adopted a predominantly linguistic or textlinguistic orientation. We focus particularly on research combining both linguistics and semiotics in order to treat verbal and graphical material as integral components of unified ac...

Semiotics has been making progressively inroads into marketing research over the past thirty years. Despite the amply demonstrated conceptual appeal and empirical pertinence of semiotic perspectives in various marketing research streams, spanning consumer research, brand communications, branding and consumer cultural studies, there has been a marke...

In this position paper, we argue that advances in intelligent cinematography require better models of the multimodal structure of filmic discourse, and of the inferences made by an audience while films are being watched. Such questions have been addressed by film scholars and cognitive scientists in the past, but their models have not so far had su...

This contribution discusses the potential role that could be played by dynamic discourse semantics as developed within formal and functional linguistic approaches to connected discourse for revitalising general semiotic approaches to complex multimodal artefacts and performances. By adopting such dynamic semantics as an integral part of a new defin...

Conceptual blending has been employed very successfully to understand the process of concept invention, studied particularly within cognitive psychology and linguistics. However, despite this influential research, within computational creativity little effort has been devoted to fully formalise these ideas and to make them amenable to computational...

The Bloomsbury Companion to M. A. K. Halliday is a comprehensive and accessible reference resource to one of the world’s leading and most influential linguists. Born in 1925, Halliday is the figure most responsible for the development of systemic functional linguistics (SFL). The impact of his work extends beyond linguistics, into the study of styl...

Leading researchers offer a range of disciplinary perspectives on the implications of spatial thinking and reasoning for education and learning. The current “spatial turn” in many disciplines reflects an emerging scholarly interest in space and spatiality as central components in understanding the natural and cultural worlds. In Space in Mind, lead...

The paper deals with the question of filling in the gutter or gap between comic panels by inferences and abductive reasoning. This is a highly discussed topic in comic studies and we choose a new and innovative way of approaching it, namely by combining linguistic, multimodal discourse analytical as well as logical and formal accounts in order to d...

Over the past four decades, discourse coherence has been studied from linguistic, psycholinguistic, computational, and applied perspectives. This volume identifies current issues and under-researched topics in the pragmatics of discourse coherence. Nine studies from various disciplines address the realization and signalling of coherence relations i...

There have been many attempts to provide accounts of visually expressed narratives by drawing on our understandings of linguistic discourse. Such approaches have however generally proceeded piecemeal --- particular phenomena appearing similar to phenomena in verbal discourse are selected for discussion with insufficient consideration of just what i...