Nicolai Marquardt

• University College London

Despite Generative AI (GenAI) systems’ potential for enhancing content creation, users often struggle to effectively integrate GenAI into their creative workflows. Core challenges include misalignment of AI-generated content with user intentions (intent elicitation and alignment), user uncertainty around how to best communicate their intents to the...

Despite Generative AI (GenAI) systems' potential for enhancing content creation, users often struggle to effectively integrate GenAI into their creative workflows. Core challenges include misalignment of AI-generated content with user intentions (intent elicitation and alignment), user uncertainty around how to best communicate their intents to the...

Chat-based prompts respond with verbose linear-sequential texts, making it difficult to explore and refine ambiguous intents, back up and reinterpret, or shift directions in creative AI-assisted design work. AI-Instruments instead embody "prompts" as interface objects via three key principles: (1) Reification of user-intent as reusable direct-manip...

There is increased interest in using generative AI to create 3D spaces for Virtual Reality (VR) applications. However, today's models produce artificial environments, falling short of supporting collaborative tasks that benefit from incorporating the user's physical context. To generate environments that support VR telepresence, we introduce SpaceB...

This paper introduces holographic cross-device interaction, a new class of remote cross-device interactions between local physical devices and holographically rendered remote devices. Cross-device interactions have enabled a rich set of interactions with device ecologies. Most existing research focuses on co-located settings (meaning when users and...

Refreshable tactile displays based on pin array technologies have a significant impact on the education of children with visual impairments, but they are prohibitively expensive. To better understand their design and the reason for the high cost, we created a database and analyzed the product architectures of 67 unique pin array technologies from l...

People with visual impairments in India have low literacy rates and only a few pursue higher education at the country's top universities. We present an insight into the educational experiences of these few university students with visual impairment based on the Frame of Interdependence. We found that educational challenges arise when interdependenc...

Two-dimensional pin array tactile displays enable access to tactile graphics that are important for the education of students with visual impairments. Due to their prohibitive cost, limited access, and limited research within HCI, the rules to design graphical primitives on these low-resolution tactile displays are unclear. In this paper, eight tac...

Many interaction techniques have been developed to best support mobile gaming – but developed gestures and techniques might not always match user behaviour or preferences. To inform this design space of gesture input for co-located mobile gaming, we present insights from a gesture elicitation user study for touch and mid-air input, specifically foc...

This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robo...

Structured note-taking forms such as sketchnoting, self-tracking journals, and bullet journaling go beyond immediate capture of information scraps. Instead, hand-drawn pride-in-craftmanship increases perceived value for sharing and display. But hand-crafting lists, tables, and calendars is tedious and repetitive. To support these practices digitall...

To better ground technical (systems) investigation and interaction design of cross-device experiences, we contribute an in-depth survey of existing multi-device practices, including fragmented workflows across devices and the way people physically organize and configure their workspaces to support such activity. Further, this survey documents a his...

This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robo...

We introduce smart vehicle proxemics, a conceptual framework for interactive vehicular applications that operationalizes proxemics to outside-the-vehicle interactions. We identify four zones around the vehicle affording different kinds of interactions and discuss the corresponding conceptual space along three dimensions (physical distance, interact...

We describe applications that implement interactions between the driver and their smart vehicle in a continuous interaction space characterized by the physical distance to the vehicle and by the smart devices that implement those interactions. Specifically, we demonstrate the principles of smart vehicle proxemics with smart rings, smartwatches, sma...

In the domain of computing education for children, much work has been done to devise creative and engaging methods of teaching about programming. However, there are many other fundamental aspects of computing that have so far received relatively less attention. This work explores how the topics of number systems and data representation can be taugh...

People’s physical movement and body language implicitly convey what they think and feel, are doing or are about to do. In contrast, current smart speakers miss out on this richness of body language, primarily relying on voice commands only. We present QUBI, a dynamic smart speaker that leverages expressive physical motion – stretching, nodding, tur...

Tangible interfaces have much potential for engendering shared interaction and reflection, as well as for promoting playful experiences. How can their properties be capitalised on to enable students to reflect on their learning, both individually and together, throughout learning sessions? This Research through Design paper describes our developmen...

Static illustrations are ubiquitous means to represent interaction scenarios. Across papers and reports, these visuals demonstrate people’s use of devices, explain systems, or show design spaces. Creating such figures is challenging, and very little is known about the overarching strategies for visually representing interaction scenarios. To mitiga...

Sketchnoting is a form of visual note taking where people listen to, synthesize, and visualize ideas from a talk or other event using a combination of pictures, diagrams, and text. Little is known about the design space of this kind of visual note taking. With an eye towards informing the implementation of digital equivalents of sketchnoting, inkin...

Computational notebooks help data analysts analyze and visualize datasets, and share analysis procedures and outputs. However, notebooks typically combine code (e.g., Python scripts), notes, and outputs (e.g., tables, graphs). The combination of disparate materials is known to hinder the comprehension of notebooks, making it difficult for analysts...

Exploring and manipulating complex virtual objects is challenging due to limitations of conventional controllers and free-hand interaction techniques. We present the TanGi toolkit which enables novices to rapidly build physical proxy objects using Composable Shape Primitives. TanGi also provides Manipulators allowing users to build objects includin...

Wearables, mobile devices and Internet-of-Things (IoT) sensors are enabling us to monitor our environment, understand our social connections, and track our personal health. However, most of these systems communicate data through information visualizations that are often ‘hidden’ inside devices, such as mobile phones and tablets, requiring users to...

There will be over 2 billion people globally who require assistive technology by 2020 however currently only 10% of people who need such technologies have access. New ways of creating interfaces which allow for sustainability for the user and the planet are essential if we are to ensure no one is left behind. This paper explores the issues of desig...

Learning through exploration is assumed to be a powerful way of introducing children to computer science concepts. However, it is uncertain how exploring physical computing toolkits can promote movement between conceptual knowledge and abstract reflection, and lead to critical thinking about technology. We investigated how children aged 9-11 years...

Sketchnoting is the process of creating a visual record with combined text and imagery of an event or presentation. Although analogue tools are still the most common method for sketchnoting, the use of digital tools is increasing. We conducted a study to better understand the current practices, techniques, compromises and opportunities of creating...

We present GazeConduits, a calibration-free ad-hoc mobile interaction concept that enables users to collaboratively interact with tablets, other users, and content in a cross-device setting using gaze and touch input. GazeConduits leverages recently introduced smartphone capabilities to detect facial features and estimate users' gaze directions. To...

Exploring and manipulating complex virtual objects is challenging due to limitations of conventional controllers and free-hand interaction techniques. We present the TanGi toolkit which enables novices to rapidly build physical proxy objects using Composable Shape Primitives. TanGi also provides Manipulators allowing users to build objects includin...

Trace figures are contour drawings of people and objects that capture the essence of scenes without the visual noise of photos or other visual representations. Their focus and clarity make them ideal representations to illustrate designs or interaction techniques. In practice, creating those figures is a tedious task requiring advanced skills, even...

Trace figures are contour drawings of people and objects that capture the essence of scenes without the visual noise of photos or other visual representations. Their focus and clarity make them ideal representations to illustrate designs or interaction techniques. In practice, creating those figures is a tedious task requiring advanced skills, even...

New physical computing toolkits offer much promise for promoting collaborative learning by engendering embodied interactions that can support collaborative discovery. To examine how these can unfold during a learning activity, we conducted a classroom study where pairs of children explored mappings between various sensors and actuators embedded in...

Designing for situational awareness could lead to better solutions for disabled people, likewise, exploring the needs of disabled people could lead to innovations that can address situational impairments. This in turn can create non-stigmatising assistive technology for disabled people from which eventually everyone could benefit. In this paper, we...

Designing interfaces or applications that move beyond the bounds of a single device screen enables new ways to engage with digital content. Research addressing the opportunities and challenges of interactions with multiple devices in concert is of continued focus in HCI research. To inform the future research agenda of this field, we contribute an...

Freehand sketching is a valuable process, input, output, and tool, often used by people to communicate and express ideas, as well as document, explore and describe concepts between researcher, user, or client. Sketches are fast, easy to create, and - by varying their fidelity - can be used in all areas of HCI. Sketching in HCI will explore and demo...

Sketching is universal. It enables us to work through problems, communicate complexity, work with people who have diverse needs, and document work processes we employ within Human-Computer Interaction. Increased interest in sketching as a methodology within HCI has led to increased attendance of interactive courses, meet-ups, and discussion groups,...

Designing for situational awareness could lead to better solutions for disabled people, likewise, exploring the needs of disabled people could lead to innovations that can address situational impairments. This in turn can create non-stigmatising assistive technology for disabled people from which eventually everyone could benefit. In this paper, we...

Although clinical best practice suggests that affect awareness could enable more effective technological support for physical rehabilitation through personalisation to psychological needs, designers need to consider what affective states matter, and how they should be tracked and addressed. In this article, we set the standard by analysing how the...

To study and understand group collaborations involving multiple handheld devices and large interactive displays, researchers frequently analyse video recordings of interaction studies to interpret people's interactions with each other and/or devices. Advances in ubicomp technologies allow researchers to record spatial information through sensors in...

The Interactions website (interactions.acm.org) hosts a stable of bloggers who share insights and observations on HCI, often challenging current practices. Each issue we'll publish selected posts from some of the leading and emerging voices in the field.

With a growing call for an increased emphasis on computing in school curricula, there is a need to make computing accessible to a diversity of learners. One potential approach is to extend the use of physical toolkits, which have been found to encourage collaboration, sustained engagement and effective learning in classrooms in general. However, li...

We contribute SurfaceConstellations, a modular hardware platform for linking multiple mobile devices to easily create novel cross-device workspace environments. Our platform combines the advantages of multi-monitor workspaces and multi-surface environments with the flexibility and extensibility of more recent cross-device setups. The SurfaceConstel...

Toolkit research plays an important role in the field of HCI, as it can heavily influence both the design and implementation of interactive systems. For publication, the HCI community typically expects toolkit research to include an evaluation component. The problem is that toolkit evaluation is challenging, as it is often unclear what 'evaluating'...

The availability of mobile device ecologies enables new types of ad-hoc co-located decision-making and sensemaking practices in which people find, collect, discuss, and share information. However, little is known about what kind of device configurations are suitable for these types of tasks. This paper contributes new insights into how people use c...

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these tex...

Hand-drawn sketches are an easy way for researchers to communicate and express ideas, as well as document, explore and describe concepts between researcher, user, or client. Sketches are fast, easy to create, and -- by varying their fidelity -- they can be used in all areas of HCI. The Applied Sketching in HCI course will explore and demonstrate th...

SurfaceConstellations is a modular hardware platform that allows users to easily create their own novel cross-device environments by assembling multiple mobile surfaces with 3D printed link modules. Our platform combines the advantages of multi-monitor workspaces and multi-surface environments with the flexibility and extensibility of more recent c...

Sketching is of great value as a process, input, output and tool in HCI, but can be confined to individual ideation or note-taking, as few researchers have the confidence to document events, studies and workshops under the public gaze. The recent surge in interest in this sometimes-overlooked skill has manifested itself in courses, workshops and li...

The availability of mobile device ecologies enables new types of ad-hoc co-located decision-making and sensemak-ing practices in which people find, collect, discuss, and share information. However, little is known about what kind of device configurations are suitable for these types of tasks. This paper contributes new insights into how people use...

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these tex...

We present RealPen, an augmented stylus for capacitive tablet screens that recreates the physical sensation of writing on paper with a pencil, ball-point pen or marker pen. The aim is to create a more engaging experience when writing on touch surfaces, such as screens of tablet computers. This is achieved by re-generating the friction-induced oscil...

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these tex...

The ability of sensing breathing is becoming an increasingly important function for technology that aims at supporting both psychological and physical wellbeing. We demonstrate ThermSense, a new breathing sensing platform based on smartphone technology and low-cost thermal camera, which allows a user to measure his/her breathing pattern in a contac...

Research in the Wild (RITW) typically involves the deployment of technology in a setting, using the methodology of `probing' contexts, to change behaviour or enhance community practice. This way of conducting HCI research is becoming an increasingly popular approach. To help in this endeavour, Rogers and Marshall [28] present an overarching framewo...

We present RealPen, an augmented stylus for capacitive tablet screens that recreates the physical sensation of writing on paper with a pencil, ball-point pen or marker pen. The aim is to create a more engaging experience when writing on touch surfaces, such as screens of tablet computers. This is achieved by regenerating the friction-induced oscill...

Public interactive displays with gesture-recognizing cameras enable new forms of interactions. However, often such systems do not yet allow passers-by a choice to engage voluntarily or disengage from an interaction. To address this issue, this paper explores how people could use different kinds of gestures or voice commands to explicitly opt-in or...

Research with interactive tabletop displays has shown much promise for collaborative scenarios. However, tabletops never became a commercial success and rarely exist outside the research community. Being relatively expensive, heavy and immobile hardware, and only limited availability of commercial applications were some of the reasons that these sy...

The ability of sensing breathing is becoming an increasingly important function for technology that aims at supporting both psychological and physical wellbeing. We demonstrate ThermSense, a new breathing sensing platform based on smartphone technology and low-cost thermal camera, which allows a user to measure his/her breathing pattern in a contac...

The ability to monitor the respiratory rate, one of the vital signs, is extremely important for the medical treatment, healthcare and fitness sectors. In many situations, mobile methods, which allow users to undertake everyday activities, are required. However, current monitoring systems can be obtrusive, requiring users to wear respiration belts o...

People are increasingly using multiple computing devices in their daily lives as portals into a shared online information space. We can select devices based on their form factor and affordances to match our task and context of use. Moreover, we are often using multiple devices at once, for example when sharing content, playing games, controlling ot...

We investigated how a physical artifact could support seamless interaction with personal activity data in everyday life. We introduce LOOP (Figure 1), a physical artifact that changes its shape according to the activity data of the owner, providing an abstract visualization. This paper reports on the design process of LOOP that was informed by inte...

Recent years have seen an increased empirical interest in designing new approaches to teaching digital fluency to wide audiences. Tangible physical computing interfaces provide much scope for teaching abstract digital fluency concepts in an engaging and playful way. However, questions remain as to how both the form factor and the corresponding task...

The ability to monitor respiratory rate is extremely important for medical treatment, healthcare and fitness sectors. In many situations, mobile methods, which allow users to undertake every day activities, are required. However, current monitoring systems can be obtrusive, requiring users to wear respiration belts or nasal probes. Recent advances...

Over the years, toolkits have been designed to facilitate the rapid prototyping of novel designs for graphical user interfaces, physical computing, fabrication, tangible interfaces and ubiquitous computing. However, although evaluation methods for HCI are widely available, particular techniques and approaches to evaluate technical toolkit research...

When designing novel user experiences, paper-pencil sketches can support the design thinking process and are valuable for communicating design ideas to others. This hands-on course will demonstrate how to integrate sketching into researchers' and interaction designers' everyday practice -- with a focus on applying the techniques to the design of no...

Real-time tracking of people's location, orientation and activities is increasingly important for designing novel ubiquitous computing applications. Top-view camera-based tracking avoids occlusion when tracking people while collaborating, but often requires complex tracking systems and advanced computer vision algorithms. To facilitate the prototyp...

HCI.Tools - Strategies and Best Practices for Designing, Evaluating and Sharing Technical HCI Toolkits. CHI 2017 Workshop Proceedings

In this workshop, we will review and discuss open issues , technical challenges and conceptual models for multi-device spatial or proxemic interaction. We aim to bring together researchers, students and practitioners working on technical infrastructures, studies and designs of spatial interfaces, or domain specific multi-device applications that us...

For digital content curation of historical artefacts, curators collaboratively collect, analyze and edit documents, images, and other digital resources in order to display and share new representations of that information to an audience. Despite their increasing reliance on digital documents and tools, current technologies provide little support fo...

The emerging Internet of Things (IoT), through which billions of everyday objects are becoming embedded with the abilities to sense their environment, compute data, and wirelessly connect to other devices, has been widely recognized as the new disruptive technology of our time. The predicted ubiquity of connected devices indicates that IoT technolo...

Learning about computing technology has become an increasingly important part of the school curriculum but it remains unclear how best to teach it to children. Here, we report on an empirical study that investigated how the process of making affects how children of different ages learn about computing technology. In one condition, they had to first...

Volunteer-driven organisations curating historic documents, such as societies and charities, often work within a bring-your-own-device (BYOD) practice and their meetings are in varying situations. A recurring challenge is finding lightweight ways to enable them to share and collectively work with documents using their own devices while in situ. We...

In this workshop, we will review and discuss challenges and opportunities for HCI in relation to cross-surface interaction in the wild based on the bring-your-own-device (BYOD) practice. We aim to bring together researchers and practitioners working on technical infra-structures for cross-surface computing, studies of cross-surface computing in par...

Internet of things (IoT) devices and sensor kits have the potential to democratize the access, use, and appropriation of data. Despite the increased availability of low cost sensors, most of the produced data is "black box" in nature: users often do not know how to access or interpret data. We propose a "human-data design" approach in which end-use...

Recent research on mobile collocated interactions has been looking at situations in which collocated users engage in collaborative activities using their mobile devices. However, existing practices fail to fully account for the culturally-dependent spatial relationships between people and their digital devices (i.e. the proxemic relationships). Bui...

The choice of consumer healthcare and wellbeing technologies has never been greater, and the introduction of consumer wearable technologies and inexpensive sensor kits means that developing bespoke personalized health devices is now possible. For example, there is a growing community making DIY diabetes technologies and the trend is spreading to ot...

The emerging Internet of Things (IoT), through which vast amounts of everyday objects are becoming embedded with computing and networking capabilities, is rapidly changing the way society uses and experiences technology. Despite this, children do not systematically learn about IoT in schools. This demonstration will showcase ConnectUs, a new IoT to...

Proxemic interaction is a vision of computing that employs proxemic relationships to mediate interaction between people and ensembles of various digital devices. In this chapter, we focus on aspects of peripheral interaction in proxemic interactions. We illustrate how to facilitate transitions between interaction outside the attentional field, the...

In this workshop, we reviewed and discussed challenges and opportunities for Human-Computer Interaction in relation to cross-surface interaction in the wild based on the bring-your-own-device (BYOD) practice. We brought together researchers and practitioners working on technical infrastructures for cross-surface computing, studies of cross-surface...

In this workshop, we reviewed and discussed opportunities, technical challenges and problems with cross-device interactions in real world interactive multi-surface and multi-device ecologies. We aim to bring together researchers and practitioners currently working on novel techniques for cross-surface interactions, focusing both on technical as wel...

In this workshop, we reviewed and discussed open issues, technical challenges and conceptual models for multi-device spatial or proxemic interaction. We brought together researchers, students and practitioners working on technical infrastructures, studies and designs of spatial interfaces, or domain specific multi-device applications that use space...

In this workshop, we will review and discuss opportunities, technical challenges and problems with cross-device interactions in interactive multi-surface and multi-device ecologies. We aim to bring together researchers and practitioners currently working on novel techniques for cross-surface interactions, focusing both on technical as well as inter...

Interactive large surfaces have recently become commonplace for interactions in public settings. The fact that people can engage with them and the spectrum of possible interactions, however, often remain invisible and can be confusing or ambiguous to passersby. In this paper, we explore the design of dynamic peripheral floor visualizations for reve...

A central issue in designing collaborative multi-surface environments is evaluating the interaction techniques, tools, and applications that we design. We often analyse data from studies using inductive video analysis, but the volume of data makes this a time-consuming process. We designed EXCITE, which gives analysts the ability to analyse studies...

When designing novel user interfaces, paper-pencil sketches can support the design thinking process and are valuable for communicating design ideas to others. In this hands-on tutorial we will demonstrate how to integrate sketching into researchers’ and interaction designers’ everyday practice – with a particular focus on the design of novel user e...