Conference PaperPDF Available

Sonapticon - space as an acoustic network

Authors:
  • Imachination Projects

Abstract and Figures

The Sonapction connects acoustics and neuroscience with space changing the Sound Dome with its 43 loudspeakers at the Center for Arts and Media (ZKM) Karlsruhe literally into a musical mindstorm. The core of the system are the so called audio neurons communicating not with electric currents but with distinct sine waves. The paper demonstrates the basic functions of such an audio neuron and how several audio neurons distributed in space create the Sonapticon turning neuronal communication mechanisms into both a sensual and a comprehensible experience. Above all the paper resumes the concert like Premiere in the context of the ZKM's IMATRONIC festival and the symposium Neuroaesthetics in November 2012.
Content may be subject to copyright.
SONAPTICON – SPACE AS AN ACOUSTIC NEURONAL NETWORK
Tim Otto Roth
Imachination Projects,
Oppenau/ Cologne, Germany
tor@imachination.net
RÉSUMÉ
The Sonapction connects acoustics and neuroscience
with space changing the Sound Dome with its 43
loudspeakers at the Center for Arts and Media ZKM
Karlsruhe literally into a musical mindstorm. [1] The
core of the system are the so called audio neurons
communicating not with electric currents but with sound
waves. This unique translation of neuronal systems’
electrical communication into acoustic communication
is combined with the deceleration of biological neuronal
processes that take place on the timescale of
milliseconds. In this way, in the Sonapticon neuronal
communication mechanisms become both a sensual and
a comprehensible experience. At the concert like
Premiere in the context of the ZKM's IMATRONIC
festival and the symposium Neuroaesthetics [2] in
November 2012 three solo piccolo players revealed by a
simple combinatoric play [3] the different facets of the
Sonapticon, to be it a microtonal sound wash or a
pseudo repetitive pattern in the inaudible sound
spectrum of ultra sound.
Figure 1. Scheme of an audio neurone.
1. INTRODUCTION – ACOUSTIC FEEDBACK
Biological neurons are interconnected by organic
wires and communicate by short electric impulses, so-
called ‘spikes’. In fig.1 you see a simple scheme of how
a neuron works: The incoming impulses are summed up
and change the membrane potential of a neuron. There
are two different types of incoming spikes: if the spike
increases the membrane potential, the sending neuron is
called excitatory, if it decreases the potential, the sender
is called inhibitory.
In the absence of external input, the membrane
potential decays to a resting potential which is marked
by the inner dashed ring. A neuron sends out an electric
impulse, it 'fires', when the incoming impulses from
other neurons force the potential over a certain
threshold, here indicated by the outer dashed ring. After
having fired the neuron is not excitable for a certain
period of time and the potential goes back to the resting
potential.
Most people do know feedback in acoustic systems
as a cascading effect of a microcophone in combination
with a loudspeaker. The Sonapticon uses a similar but
different scheme. The basic units of the Sonapticon are
audio neurons interconnected not by wires but by sound
transmitted in space. An audio neuron registers impulses
(spikes) from its acoustic environment by a microphone
and fires its own impulse by a loudspeaker (fig.1). The
clue of that system is that every neuron gets assigned a
sine tone with an individual frequency.
Figure 2: Visualization of the Python based
audio-neuron showing the frequency tracking
(top), the current membrane potential (circle in
the middle) and the history plot of the membrane
potential (bottom).
Looking to a sound spectrum you see that most
sounds do consist of a broad band of frequencies. But
sine tones oscillating at a single frequency show up
characteristic peaks in a spectrum (see top of fig. 2).
This is how you can simply connect the audio neurons:
Every audio neuron registers a specific set of
frequencies assigned to other neurons. These
frequencies are marked as lines in two colours in the
spectrum. If an audio neuron registers a significant steep
peak at one of these lines it interprets this as impulse of
a connected neuron and the line flashes. Depending on
the colour assigned to a line an impulse has an
excitatory or inhibitory effect.
2. DECELERATION OF BIOLOGICAL
PROCESSES
Unlike in in vivo neural tissue, where neuronal
interactions occur on the timescale of milliseconds, the
Sonapticon allows to decelerate all biological processes
by the factor 10-200. This generates an environment
where spectators can identify and understand intuitively
the causal relationship between integration and initiation
of spikes on a single neuron level. Thereby, the
Sonapticon creates an unprecedented sensual experience
of the basic computational principles that underlie all
mental processes, be it the initiation of movement, the
creation of simple thoughts, or even the emergence of
consciousness.
This simple interaction scheme is the basis of all
nervous activity. It is still an enigma how these
interactions form something like a simple thought in our
brain. With latest microscopic techniques you can
observe in vivo in small probes of neuronal tissues how
firing neurons self-organize and create synchronized
activity patterns clip. Unlike in in vivo neural tissue,
where neuronal interactions occur on the timescale of
milliseconds, the in silico technique of the Sonapticon
allows to decelerate all biological processes. This
generates an environment where spectators can identify
and understand intuitively the causal relationship
between integration and initiation of spikes on a single
neuron level.
Consequently the Sonapticon is a system of
decelerated synthetic neurons which allows a real time
interaction with neuronal dynamics and control of the
biological parameters. The Sonapticon combines two
basic components:
- digital in silico methods, with latest mathematical
models of biological neurons (e. g. the conductivity
based model by Alain Destexhe) [4]
and
- an empirical environment (the Sound Dome) using
acoustics as an analogue space of experimentation and
interaction. Here delay effects come into play due to the
propagation speed of sound resulting in dynamics which
differ essentially from cable wired systems.
Principally every computing device with a
microphone and loudspeaker can function as an audio
neuron to be it a laptop, a tablet computer or a smart
phone. In August 2012 a first performance did take
place at the Bernstein Centre for Computational
Neuroscience on the Campus of the Charité in Berlin.
The visitors installed a little python based software (see
screenshot fig. 2) to be it for PC or Mac and changed
their laptops with small external speakers into acoustic
neurons. Step by step a network of twenty acoustic
neurons was built up, exploring the changing dynamics
by adding further neurons.
Above all the Sonapticon invites to interact with the
system in all different forms. An instrument like the
singing saw producing clear sine peaks resulted to be a
perfect device to explore the system's resonances.
3. THE SONAPTICON AT THE ZKM'S SOUND
DOME
Thanks to the invitation by the Institute of Music and
Acoustics at the Center for Art and Media, ZKM
Karlsruhe we – the author and the neuro-mathematician
Benjamin Staude – could explore together the
Sonapticon as guest composers from 2010-12 in a very
sophisticated environment. In collaboration with the
sound engineer Holger Stenschke from ZKM we were
changing the Sound Dome into an acoustic neuronal
network. The Sound Dome consists of 43 loudspeakers
building an impressing hemisphere of 15 meters in
diameter. The loudspeakers get joined via microphones.
Intuitive understanding of the acoustic feedback through
sine tones in the Sound Dome space is complemented
visually through LED lights mounted on the speakers
flashing up in a blue or yellow colour in case the neuron
is firing. Above all a video projection on the floor
shows a visualization of all the active neurons with their
changing membrane potentials.
Figure 3: Impression from the premiere with
three piccolo players standing around the floor
projection and flashing loudspeakers in the
background.
In the ZKM's Sound Dome the audio analysis uses a
frame work based on MaxMsp programmed by Holger
Stenschke. The adaptation of the neuron model and the
composition frame work was realized in Python by
Benjamin Staude. For the visualization Tim Otto Roth
used Gem in combination with Puredata.
4. SPACE
First of all the Sonapticon creates in the Sound Dome
a completely new experience emerging the visitor into a
surrounding acoustic atmosphere: You feel to stand in
the middle of an organic process of self organizing
tones.
Above all the use of sine tones assigned to specific
loudspeaker in the space creates a particular effect. If
you are walking around in the installation the sine tone
patterns recompose differently in the visitors ears
depending on the locally dominant tones. If the system
runs very fast you even get the effect of standing waves
changing even if you slightly turn the head.
Figure 3: Impression from the audio visual
spatial setting in the Sound Dome (Klangdom) at
the ZKM Karlsruhe.
Finally this controlled environment allows to explore
systematically the characteristics of the Sonapticon. For
instance with regards to the size of the Sound Dome and
the speed how sound waves propagate in space delay
effects come into play which appear also in neuronal
tissues: So here it makes a difference where and when a
sound is played in the environment – an invitation for
musicians to play with the environment.
5. A NEW INSTRUMENT
Thus, we transformed with the Sonapticon the Sound
Dome into a new instrument for which we have
developed a special method of composing in and with
the Sound Dome space: choosing scales up to the eighth
tone range, giving tones a location by assigning
frequencies to an audio neuron, setting the connections
between the audio neurons and finally varying the tempi
by playing with the system's update rate.
The piece at the concert premiere comprised eight
movements starting with a single neuron. Step by step
the number of neurons the three piccolo players
interacted with was increased. Above all the
composition used different tonalities as a Western
chromatic, an Indian Shruti or a microtonal eighth tone
scale. In the last movements the frequencies were
transposed subsequently by perfect fifths ending up in a
finale with inaudible ultra sound.
Figure 4. Compostion matrix with connection
map at the right and the resulting network at the
left.
Finally the models themselves are a playfield for
composition. Parameters can be changed globally, but
also the characteristics of individual neurons can be
varied. Modifying the ratio of neurons and interneurons
has a significant effect. Until now this was studied only
rudimentary, so it is a future task for the Sonapticon to
explore also with different models acoustically the
phenomenon of synaptic plasticity.
6. REFERENCES
[1] An extended documentation of the project including
two video introductions is provided by the project
page: http://www.pixelsex.org/sonapticon.
[2] Record of Tim Otto Roth's and Benjamin Staude's
talk on the Neuroaesthetics symposium at ZKM
Karlsruhe: https://itunes.apple.com/de/itunes-
u/neuroaesthetics-symposium/id601309561?mt=10 .
[3] Only one tone is played by each player in a certain
sequence, so together the three soloists play all
possible tone combinations. This time based
combinatory play is inspired by Tom Johnson's
counting pieces. See: http://www.editions75.com .
[4] Destexhe, Alain: Self-sustained asynchronous
irregular states and Up–Down states in thalamic,
cortical and thalamocortical networks of nonlinear
integrate-and-fire neurons, in: Journal of
Computational Neuroscience 27 (Dec 2009) Nr. 3, p.
493-506. Destexhe's model has been adapted by
adding a refractory period.
[5] Barry, Robert: Artist collaborates with neuroscientist
to build 'audio-neurons', interview on wired.co.uk
(10 December 2012),
http://www.wired.co.uk/news/archive/2012-
12/10/tim-otto-roth .
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Randomly-connected networks of integrate-and-fire (IF) neurons are known to display asynchronous irregular (AI) activity states, which resemble the discharge activity recorded in the cerebral cortex of awake animals. However, it is not clear whether such activity states are specific to simple IF models, or if they also exist in networks where neurons are endowed with complex intrinsic properties similar to electrophysiological measurements. Here, we investigate the occurrence of AI states in networks of nonlinear IF neurons, such as the adaptive exponential IF (Brette-Gerstner-Izhikevich) model. This model can display intrinsic properties such as low-threshold spike (LTS), regular spiking (RS) or fast-spiking (FS). We successively investigate the oscillatory and AI dynamics of thalamic, cortical and thalamocortical networks using such models. AI states can be found in each case, sometimes with surprisingly small network size of the order of a few tens of neurons. We show that the presence of LTS neurons in cortex or in thalamus, explains the robust emergence of AI states for relatively small network sizes. Finally, we investigate the role of spike-frequency adaptation (SFA). In cortical networks with strong SFA in RS cells, the AI state is transient, but when SFA is reduced, AI states can be self-sustained for long times. In thalamocortical networks, AI states are found when the cortex is itself in an AI state, but with strong SFA, the thalamocortical network displays Up and Down state transitions, similar to intracellular recordings during slow-wave sleep or anesthesia. Self-sustained Up and Down states could also be generated by two-layer cortical networks with LTS cells. These models suggest that intrinsic properties such as adaptation and low-threshold bursting activity are crucial for the genesis and control of AI states in thalamocortical networks.
Artist collaborates with neuroscientist to build 'audio-neurons', interview on wired
  • Robert Barry
Barry, Robert: Artist collaborates with neuroscientist to build 'audio-neurons', interview on wired.co.uk (10 December 2012), http://www.wired.co.uk/news/archive/201212/10/tim-otto-roth.
s and Benjamin Staude's talk on the Neuroaesthetics symposium at ZKM Karlsruhe: https
  • Tim Otto Record
  • Roth
Record of Tim Otto Roth's and Benjamin Staude's talk on the Neuroaesthetics symposium at ZKM Karlsruhe: https://itunes.apple.com/de/itunes- u/neuroaesthetics-symposium/id601309561?mt=10.