Tuomo Hiippala

Tuomo Hiippala
University of Helsinki | HY · Department of Languages

PhD

About

59
Publications
42,321
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
1,177
Citations
Citations since 2016
47 Research Items
1126 Citations
2016201720182019202020212022050100150200250300
2016201720182019202020212022050100150200250300
2016201720182019202020212022050100150200250300
2016201720182019202020212022050100150200250300
Introduction
I do research on multimodality, that is, how natural language, photographs, drawings, diagrams, layout and other forms of communication interact and co-operate with each other in different communicative situations. My approach to multimodality is mainly informed by linguistics, but I have also explored the use of computational approaches to scale up the volume of data. At the moment, my research focuses on the automatic analysis of multimodal content in social media.
Additional affiliations
January 2017 - December 2017
University of Helsinki
Position
  • PhD Student
April 2015 - December 2016
University of Jyväskylä
Position
  • PostDoc Position
February 2014 - June 2014
University of Helsinki
Position
  • Post-doctoral researcher (personal grant)
Education
January 2008 - January 2014
University of Helsinki
Field of study
  • English Philology
September 2002 - September 2007
University of Helsinki
Field of study
  • English Philology

Publications

Publications (59)
Book
Full-text available
This textbook provides the first foundational introduction to the practice of analysing multimodality, covering the full breadth of media and situations in which multimodality needs to be a concern. Readers learn via use cases how to approach any multimodal situation and to derive their own specifically tailored sets of methods for conducting and e...
Book
Full-text available
This book develops a new framework for describing the structure of multimodal documents: how language, image, layout and other modes of communication work together to convey meaning. Building on recent research in multimodal analysis, functional linguistics and information design, the book examines the textual, visual, and spatial aspects of page-b...
Article
Full-text available
Digital longform journalism has recently attracted increased attention among both academics and professionals. This study contributes to the growing body of research by dissecting the multimodal structure of digital longform journalism, that is, how the emerging genre combines written language, photography, short videos, maps and other graphical el...
Chapter
Full-text available
As the use and diversity of diagrams across many disciplines grows, there is an increasing interest in the diagrams research community concerning how such diversity might be documented and explained. In this article, we argue that one way of achieving increased reliability, coverage, and utility for a general classification of diagrams is to draw o...
Article
Full-text available
Globalization, urbanization and international mobility have led to increasingly diverse urban populations. Compared to traditional traits for measuring urban diversity, such as ethnicity and country of origin, the role of language remains underexplored in understanding diversity, interactions between different groups and socio-spatial segregation....
Article
Full-text available
Sport and exercise contribute to health and well-being in cities. While previous research has mainly focused on activities at specific locations such as sport facilities, "in-formal sport" that occur at arbitrary locations across the city have been largely neglected. Such activities are more challenging to observe, but this challenge may be address...
Article
Full-text available
Identifying users’ place of residence is an important step in many social media analysis workflows. Various techniques for detecting home locations from social media data have been proposed, but their reliability has rarely been validated using ground truth data. In this article, we compared commonly used spatial and spatio-temporal methods to dete...
Article
Full-text available
In this article, we argue for the benefits of combining large-scale analyses of visual materials currently pursued within digital humanities with insights from multimodality research, which is an emerging discipline that studies how human communication relies on appropriate combinations of expressive resources. We show that concepts developed withi...
Conference Paper
Full-text available
This system demonstration paper describes ongoing work on a tool for fair and reproducible use of paid crowdsourcing in the digital humanities. Paid crowdsourcing is widely used in natural language processing and computer vision, but has been rarely applied in the digital humanities due to ethical concerns. We discuss concerns associated with paid...
Chapter
Full-text available
In this chapter our aim is to develop a corpus-based procedure for empirical studies of style in corporate logos.We present a data-driven corpus study of n=50 logos, 25 each from the oil industry and non-governmental environmental organizations (NGOs), in order to see if we can pick up significant differences in style between the two groups.We use...
Article
Full-text available
This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural sciences, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowdsourced...
Method
tweetsearcher is a Python tool designed for downloading tweets from Twitter with Academic Research credentials. It downloads and automatically parses the json response from Twitter's API v2 saving it to a pickled dataframe. If one is inclined to do so, there's another script file included which will turn the pickled dataframes into geopackage files...
Article
Full-text available
This article discusses the prospects and challenges of combining multimodality theory with distant viewing, a recent framework proposed in the field of digital humanities. This framework advocates the use of computational methods to enable large-scale analysis of visual and multimodal materials, which must be nevertheless supported by theories that...
Preprint
Full-text available
This contribution describes a two-course module that seeks to provide humanities majors with a basic understanding of language technology and its applications using Python. The learning materials consist of interactive Jupyter Notebooks and accompanying YouTube videos, which are openly available with a Creative Commons licence.
Preprint
Full-text available
In this article, we bring together theories of multimodal communication and computational methods to study how primary school science diagrams combine multiple expressive resources. We position our work within the field of digital humanities, and show how annotations informed by multimodality research, which target expressive resources and discours...
Article
Full-text available
Understanding the activities and preferences of visitors is crucial for managing protected areas and planning conservation strategies. Conservation culturomics promotes the use of user-generated online content in conservation science. Geotagged social media content is a unique source of in situ information on human presence and activities in nature...
Article
Full-text available
Twitter is a popular social media platform for scholarly research, because the user-generated content on the platform can also include geographic and temporal information. We collect a corpus of 38 million Twitter messages with two million geographical coordinates to map the languages used across Finland at the level of regions and municipalities....
Article
Full-text available
Parks and other green spaces are an important part of sustainable, healthy and socially equal urban environment. Urban planning and green space management benefit from information about green space use and values, but such data are often scarce and laborious to collect. Temporally dynamic geographic information generated by different mobile devices...
Chapter
Full-text available
Today we are witnessing an increased use of data visualization in society. Across domains such as work, education and the news, various forms of graphs, charts and maps are used to explain, convince and tell stories. In an era in which more and more data are produced and circulated digitally, and digital tools make visualization production increasi...
Preprint
Full-text available
In this article, we propose a multimodal perspective to diagrammatic representations by sketching a description of what may be tentatively termed the diagrammatic mode. We consider diagrammatic representations in the light of contemporary multimodality theory and explicate what enables diagrammatic representations to integrate natural language, var...
Preprint
Full-text available
Although the number of articles in visual and multimodal communication that include statistical validation of claimed results is increasing, we suggest in this article that this is by no means enough. Statistical methods should belong to every multimodality researcher's toolset precisely because the phenomena under study are subtle and complex. Wit...
Preprint
Full-text available
This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural science, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowd-sourced...
Preprint
Full-text available
This article compares two multimodal resources that consist of diagrams which describe topics in elementary school natural sciences. Both resources contain the same diagrams and represent their structure using graphs, but differ in terms of their annotation schema and how the annotations have been created - depending on the resource in question - e...
Article
Full-text available
This article explores the linguistic landscape of social media posts associated with specific geographic locations using computational methods. Because physical and virtual spaces have become increasingly intertwined due to location-aware mobile devices, we propose extending the concept of linguistic landscape to cover both physical and virtual env...
Article
Full-text available
Improved understanding of human-nature interactions is crucial to conservation science and practice, but collecting relevant data remains challenging. Recently, social media have become an increasingly important source of information on human-nature interactions. However, the use of advanced methods for analysing social media is still limited, and...
Conference Paper
Full-text available
Social media contains a wealth of information about human activities in different places. This infor- mation can complement data collection efforts in resource-scarce fields such as nature conservation. However, social media platforms differ in popularity, content, and access to data, and the choice of platform may greatly affect the resulting analysis....
Presentation
Illegal wildlife trade is one of the biggest threats to biodiversity conservation, as many species, including iconic species such as rhinoceros and elephant taxa, are targeted for their meat, trophies and other body parts. Over the last years, the scale and nature of illegal wildlife trade has changed dramatically. The Internet is becoming a major...
Conference Paper
Full-text available
This paper describes ongoing work on a multimodal resource based on the Allen Institute AI2 Diagrams (AI2D) dataset, which contains nearly 5000 grade-school level science diagrams that have been annotated for their elements and the semantic relations that hold between them. This emerging resource, named AI2D-RST, aims to provide a drop-in replaceme...
Article
Full-text available
Article impact statement: Machine learning can be used to monitor and assess the extent of illegal wildlife trade on social media platforms.
Article
Full-text available
This review article provides an overview of the research conducted within the Genre and Multimodality framework, which has been used to describe the multimodality of page-based documents and other multimodal artefacts over the past 15 years. The article explicates the motivation and inspiration for developing the framework, introduces its central t...
Conference Paper
Full-text available
This paper presents a system that uses machine learning to recognize military vehicles in social media images. To do so, the system draws on recent advances in applying deep neural networks to computer vision, while also making extensive use of openly available libraries, models and data. Training a vehicle recognition system over three classes, th...
Article
Full-text available
This article examines the concepts of agency, transformation and transduction in the context of document design. These concepts have been previously used to describe communicative actions and sign-making among individuals: whereas agency focuses on the individual’s capabilities as a sign-maker, transformation and transduction describe how individua...
Article
Full-text available
This article summarises the results of my doctoral dissertation, which studied how the tourist brochures combine language, images, layout, and maps to promote the destination and to guide the reader. The data consisted of the English-language tourist brochures published by the city of Helsinki between 1967 and 2008, which were annotated for their c...
Thesis
Full-text available
This dissertation studied the structure of multimodal artefacts, or how language, image and other semiotic modes combine and interact in documents. This places the study within the emerging field of multimodal research, which uses linguistic methods to study the interaction of multiple semiotic modes. Despite the growing amount of multimodal resea...
Article
Full-text available
This research-in-progress report describes ongoing work on a doctoral dissertation, which attempts to model the prototypical structure of the tourist brochure as a multimodal artefact. By using a multimodal corpus based on the Genre and Multimodality model, the dissertation investigates how the brochures use both language and image to fulfil their...
Chapter
Full-text available
The concept of localisation is typically associated with the field of information technology and software development, and involves describing the process of translating a product and its documentation, verifying the translation, and accounting for any cultural factors that might be related to the use of the product. Localisation has emerged as a r...
Article
Full-text available
This paper argues that the concept of a reading path in multimodal research can be improved by previous research on visual perception in psychology and brain sciences, and particularly by the work done within eye-tracking studies. The paper argues that in its current state, the concept of a reading path is not sufficiently reliable due to the lack...
Conference Paper
Full-text available
This paper aims to show how the Genre and Multimodality (GeM) model (Bateman 2008) can be applied to the study of multimodal meaning-making and genre as higher level organizational concept in print media. The theoretical framework of the GeM model combines the methodologies of corpus and text linguistics and document design in a multi-layered and c...
Thesis
Full-text available
This thesis analyses tourist brochures as a genre of print media, focusing on the city of Helsinki, the capital of Finland. Drawing on the systemic-functional theory of language by Halliday (1978, 1994), the study conducts a multisemiotic discourse analysis of the collected materials, focusing on language, image, colour, composition and typography....

Network

Cited By

Projects

Projects (2)
Project
The Institute on Multimodality provides a consolidated perspective on the theoretical, methodological, and practical understanding of multimodality research. The institute will broadly –through its multidisciplinary faculty and technical programme– present an all encompassing and outward-looking view of the emerging discipline of multimodality research at the interface of Minds, Media, and Technology. Here, the scientific agenda of the institute will encompass the relationships between multimodality studies across areas such as cognition, artificial intelligence, communications and media, data science, visuo-auditory media, and interaction studies. The institute hosts an invited faculty delivering lectures, intensive tutorials, and keynotes; opportunities also exist for young researchers to position ongoing / early stage research, discuss, and network with institute faculty and participants. http://codesign-lab.org/institute2022/