Joseph Malloch

Dalhousie University | Dal · Faculty of Computer Science

Immersive augmented reality (AR) is an exciting medium for locative narrative. Immersive AR experiences are largely custom-made and site-specific, however: we lack generic tools that help authors consider interactions between physical layout, viewer perspective, and story progression, for specific sites or for locations unknown to the author. In th...

EXtended Reality (XR) is a rapidly developing paradigm for computer entertainment, and is also increasingly used for simulation, training, data analysis, and other non-entertainment purposes, often employing head-worn XR devices like the Microsoft HoloLens. In XR, integration with the physical world should also include integration with commonly use...

EXtended Reality (XR) is a rapidly developing paradigm for computer entertainment, and is also increasingly used for simulation, training, data analysis, and other non-entertainment purposes, often employing head-worn XR devices like the Microsoft HoloLens. In XR, integration with the physical world should also include integration with commonly use...

With the rapidly growing interest in AR, grows the motivation to overcome some of the problems facing the implementations of the technology. The main challenge encountered in the building of large-scale mixed reality AR games is the uniqueness of the spatial settings in which the game will be experienced by the user. Game designers will require dat...

Museums are constantly searching for new ways to increase engagement with their exhibits, from electronic guides to modern digital technologies such as special-purpose tablets, smartphones, and virtual and augmented reality (AR). For AR exhibits in particular, promoting shared experience and group cohesion is not straightforward. In this work, we i...

In recent years, museums have embraced the use of digital technologies to add interactivity to exhibits. New tools such as wireless beacons, QR codes and markerless trackers paired with powerful smartphones are used to implement applications ranging from guides that provide supplementary material as web pages or audio to spatially precise augmented...

This chapter discusses possible links between the fields of computer music (CM) and human-computer interaction (HCI), particularly in the context of the MIDWAY project between Inria, France and McGill University, Canada. The goal of MIDWAY is to construct a “musical interaction design workbench” to facilitate the exploration and development of new...

The Digital Orchestra Toolbox for Max is an open-source collection of small modular software tools for aiding the development of Digital Musical Instruments. Each tool takes the form of an “abstraction” for the visual programming environment Max, meaning it can be opened and under- stood by users within the Max environment, as well as copied, modif...

This paper presents the work to maintain several copies of the digital musical instrument (DMI) called the T-Stick in the hopes of extending their useful lifetime. The T-Sticks were originally conceived in 2006 and 20 copies have been built over the last 12 years. While they all preserve the original design concept, their evolution resulted in vari...

Gaming with large displays can promote whole body interaction, social gameplay among acquaintances and strangers alike, and foster a sense of immersion. We present SpaceHopper, a large-scale, floor-projected version of the game Asteroids, where players bounce on a hopper ball to control their ship. We implemented two interaction modes: bounce to re...

We articulate a need for the representation of temporal objects represented by dynamic, short-lived mapping connections instantiated from a template, in tools for designing and using interactive media systems. A list of requirements is compiled from examination of existing tools, practical use cases, and abstract considerations of node connectivity...

Although users accomplish ever more tasks on touch-enabled mobile devices, gesture-based interaction remains limited and almost never customizable by users. Our goal is to help users create gestures that are both personally memorable and reliably recognized by a touch-enabled mobile device. We address these competing requirements with two dynamic g...

In this paper, we explore how people use touchscreens to express emotional intensity, and whether these intensities can be understood by oneself at a later date or by others. In a controlled study, 26 participants were asked to express a set of emotions mapped to predefined gestures, at range of different intensities. One week later, participants w...

While several researchers have grappled with the problem of comparing musical devices across performance, installation, and related contexts, no methodology yet exists for producing holistic, informative visualizations for these devices. Drawing on existing research in performance interaction, human-computer interaction, and design space analysis,...

This paper describes a versioning and annotation system for supporting collaborative, iterative design of mapping layers for digital musical instruments (DMIs). First we de- scribe prior experiences and contexts of working on DMIs that has motivated such features in a tool, describe the current prototype implementation and then discuss future work...

Gesture-typing is an efficient, easy-to-learn, and errortolerant technique for entering text on software keyboards. Our goal is to "recycle" users' otherwise-unused gesture variation to create rich output under the users' control, without sacrificing accuracy. Experiment 1 reveals a high level of existing gesture variation, even for accurate text,...

Music is an evolutionarily deep-rooted, abstract, real-time, complex, non-verbal, social activity. Consequently, interaction design in music can be a valuable source of challenges and new ideas for HCI. This workshop will reflect on the latest research in Music and HCI (Music Interaction for short), with the aim of strengthening the dialogue betwee...

Moving new DMIs from the research lab to professional artistic contexts places new demands on both their design and manufacturing. Through a discussion of the Prosthetic Instruments, a family of digital musical instruments we designed for use in an interactive dance performance, we discuss four different approaches to manufacturing – artisanal, bui...

We introduce libmapper, an open source, cross-platform software library for flexibly connecting disparate interactive media control systems at run-time. This library implements a minimal, openly-documented protocol meant to replace and improve on existing schemes for connecting digital musical instruments and other interactive systems, bringing cla...

We present libmapper, a software library and protocol for providing network-enabled discovery and connectivity of real-time control signals. Today there is a trade-off present in the state of the art for music-related networking. At one extreme, we have many systems still using MIDI, an old and insufficient standard for specifying keyboard-oriented...

In this paper we present an approach to instrument augmentation using the musician's ancillary gestures to enhance the liveliness of real-time digitally processed sound. In augmented instrument praxis, the simultaneous control of the initial instrument and its ' electric/electronic extension is a challenge due to the musician's physi...

This paper presents ongoing work into the design of interfaces for live musical performance using gesture controlled sound spatialization. In particular, we dis-cuss the use of performer's gestures to allow them to manipulate spatialization parameters without their con-scious control. A number of motion capture sessions in-volving a cello performer...

We present a live solo concert performance of an original piece of music - Everybody to the power of one - written for the soprano T-Stick digital musical instrument. Like other digital musical instruments, the T-Stick enables the reincorporation of performer gesture as the main source of control in computer-based music making. A brief description...

SenseStage is a research-creation project to develop a wireless sensor network infrastructure for live per-formance and interactive, real-time environments. The project is motivated by the economic and tech-nical constraints of live performance contexts and the lack of existing tools for artistic work with wire-less sensing platforms. The developme...

This paper introduces the CIRMMT/ McGill Digital Orches- tra Project, the objective of which was to develop new cre- ative resources for the integration of digital technologies and live musical performance. The authors present five new Dig- ital Musical Instruments (DMIs) designed during the project, and discuss issues surrounding the creation of r...

In this paper we present an approach to instrument augmentation using the musician's ancillary gestures to enhance the liveliness of real-time digitally processed sound. In augmented instrument praxis, the simultaneous control of the initial instrument and its' electric/electronic extension is a challenge due to the musician's physical and psycholo...

The paper describes the development of the hyper-kalimba, an augmented instrument created by the authors. This de-velopment was divided into several phases and was based on constant consideration of technology, performance and compositional issues. The basic goal was to extend the sound possibilities of the kalimba, without interfering with any of...

The SenseWorld DataNetwork framework addresses the is-sue of sharing and manipulating multiple data streams among different media systems in a heterogenous interactive per-formance environment. It is intended to facilitate the cre-ation, rehearsal process and performance practice of collab-orative interactive media art works, by making the sharing...

An experimental study was performed on the effects of the visibility of a performer’s gestures on the identification of virtual sound trajectories in the concert hall. We found that when working in synchrony, the performer’s gestures integrate with the audio cues to significantly increase identification performance, normalize for the effects of off...

This paper summarises a panel discussion at the 2007 International Computer Music Conference on movement and gesture data formats, presents some of the formats currently in development in the computer music community, and outlines some of the challenges involved in future development.

This paper describes the design and implementation of a framework designed to aid collaborative development of a digital musical instrument mapping layer. The goal was to create a system that allows mapping between controller and sound parameters without requiring a high level of technical knowledge, and which needs minimal manual intervention for...

The last decade has seen the development of standards for music notation (MusicXML), audio analysis (SDIF), and sound control (OSC), but there are no widespread standards, nor structured approaches, for handling music-related movement, action and gesture data. This panel will address the needs for such formats and standards in the computer music co...

This paper presents the development of methods for gesture control of sound spatialization. It provides a comparison of seven popular software spatialization systems from a control point of view, and examines human-factors issues relevant to gesture control. An effort is made to reconcile these two design- and parameter-spaces, and draw useful conc...

This paper describes the design and implementation of a system for collaborative development of a digital musical instrument mapping layer. System development included the design of a decentralized network for the management of peer-to-peer data connections using Open Sound Con-trol. A graphical user interface for dynamically creating, modifying, a...

This paper describes the T-Stick, a new family of digital musical instruments. It presents the motivation behind the project, hardware and software design, and presents insights gained through collaboration with performers who have col-lectively practised and performed with the T-Stick for hun-dreds of hours, and with composers who have written pie...

This paper describes an online Wiki, a collaborative Web site designed to allow users to edit and add content. It was created at the Input Devices and Music Interaction Laboratory with the aim of promoting and supporting the construction of new musical interfaces. Although many individual universities and research centres offer sources of relevant...

This paper presents a novel digital musical instrument using an interface constructed from a large spring that offers natural kinesthetic feedback through its inherent stiffness. The design of the first prototype is described and discussed in relation to the notion of effort in musical performance. Audio feedback and distortion are introduced as a...

This paper describes the adaptation of an existing model of hu- man information processing for the categorization of digital musi- cal instruments in terms of performance context and behavior. It further presents a visualization intended to aid the analysis of ex- isting DMIs and the design of new devices. Three new interfaces constructed by the au...

This paper describes the adaptation of an existing model of hu- man information processing for the categorization of digital musi- cal instruments in terms of performance context and behavior. It further presents a visualization intended to aid the analysis of ex- isting DMIs and the design of new devices. Three new interfaces constructed by the au...

This paper describes an adaptation and implemen-tation of existing beat-induction algorithms for use as a real-time control parameter of an interactive music system. Although beat tracking has been implemented for various purposes, approaches have focused on traditional western music in which the tempo is generally constant. Leveraging existing bea...

While several researchers have grappled with the problem of comparing musical devices across performance, installa- tion, and related contexts, no methodology yet exists for producing holistic, informative visualizations for these de- vices. Drawing on existing research in performance inter- action, human-computer interaction, and design space anal...