Playing With Pixels

Full text

Playing with pixels Joan Soler-Adillon
www.joan.cat
Published in1:
Soler-Adillon, Joan (2011), “Playing with Pixels”, in Impresión Expandida / Expanded
Print, eds. Eloi Puig, Alicia Vela and Antonia Vilà (Barcelona, Universitat de
Barcelona, Ministerio de Ciencia e Innovación and European Regional Development Fund,
ISBN: 978-84-614-8786-8), 169- 176.
Translation by Alex Reynolds.
1. Introduction
I started working with pixels whilst studying with one of those
professors one remembers fondly for having been a great source of
inspiration. In fact, this story ends with another unforgettable
professor. But let's take it one step at a time.
With Daniel Rozin, I learned to look at and work with digital images,
one pixel at a time (or, as the course was named: pixel by pixel). The
principle is in fact quite simple: if every image is so many pixels
wide and so many pixels high, and each pixel contains more than a
single colour value, we can examine pixel by pixel in order to
analyse, change, or play with it... and by doing this we can virtually
create any possible image.
From this point of view, a digital image is, regardless of its origin,
nothing more than numerical data. It can be generated by a camera, a
computer, or can be one of the twenty-five images per second that make
up a video. All these images are coded in the same way, and can
therefore be handled in the same way.
For an artist, this is a fascinating fact. It doesn't matter whether
it is a still or moving, synthetic or real image: through programming,
it turns into an immensely rich creative medium. Programming isn't
easy, but it isn't beyond anyone's reach either. After all,
programming is reduced to a relatively small series of logical and
mathematical operations that even a child in his last primary school
years could learn.
In fact, it is surprising how we are constantly using programmable
devices these days, and yet programming the majority of them remains
not only technically impossible, but also undecipherable to almost
everyone. Think of any electronic phone, or a domestic television set.
Why isn't it possible to reprogram my remote control to make it carry
out the tasks that are most convenient for me, or to lock my phone by
pressing a sequence of keys of my choice?
In the 70s, Alan Kay led the Xerox Alto project, which was a direct
precursor of today's personal computers. It allowed for the
1
Joan Soler-Adillon, Playing with Pixels

possibility of using a simple programming language, which is one of
the activities that groups of children visiting the Xerox PARC would
do: they would programme their own applications. The very name of the
programming language they used, Smalltalk, already suggested that Kay
thought about programming as something that needn't be in the hands of
just a handful of people.
Living in a culture that many consider to be 'digital', we should
perhaps begin to question whether the ability to program a computer
(which is also an excellent way to learn about logic, maths and
geometry) shouldn't be universal. Even though the opinion isn't a
mainstream position, within the art world there is already a huge
group of people who have used programming and digital media as just
another artistic tool for a long time, and many have decided to use it
as their main medium.
The present text is a personal, descriptive account of how I entered
the digital art world through programming, understood from the
insignificant position of a programmer who isn't a proper programmer.
2. Liquid Video
Liquid Video is a piece that came about as a result of working with
pixels in the way I described above. It is, at the same time, a modest
example of something that is also related to the last piece that I
will present within this text: the concept of emergence, which
describes complex phenomena generated by repeating simple rules and
processes2.
In fact, Liquid Video isn't really an interactive work. The term
'reactive' doesn't define it correctly either. The most appropriate
term to describe it would be that of a live painting, or a mirror. It
is a work that converses with its environment3 regardless of whether
the environment contains a human presence or not. In this way, the
viewer can choose whether to participate in the constant creation of
the image that develops within it or not. If he does, he has to adapt
to the rules and rhythm of the piece, as its process will continue
regardless of the image being read by the camera.
The installation works as a continuous loop; a coming and going
between two states that can be defined, from their appearance, as
liquid and drying states. The piece consists of a projection screen
and a video camera that feeds the system with the images it works
with. In this way, always using images from the video, the first state
presents the visitor with a reflection that is increasingly distorted.
The colours smear along the digital canvas as if a 'water effect' had
been applied to them until shapes are unrecognisable, and the only
recognisable features that remain are the actual colours of the
environment and the viewer.
2
Joan Soler-Adillon, Playing with Pixels

Liquid Video: 'Water Effect'
At this point, the system enters the drying state, and the image from
the video gradually pulls itself together again. The colours slowly
return to their original place, but leave small traces and strokes
behind them along the way. In this way, the resulting effect is
similar to that of a painting drying, which is created by using the
image of the mirror that emulates the video. Just when the image
begins to look like a simple replica of what we would see through a
conventional video, the process of the water begins again.
Liquid Video: Drying state
3
Joan Soler-Adillon, Playing with Pixels

The process that creates these two effects is actually extremely
simple. The system is only reading a limited number of pixels that
make up the image of the original video (just 10,000 out of a total of
307,200), but instead of painting them where it should, it places them
on a point that is generated by a basic programming concept: the
Random Walker.
The Random Walker is nothing other than an element whose position is
constantly moving up or down, or right or left, randomly. It is a
typical exercise in introductory programming courses. This is
therefore the foundation of the piece: to read pixels in one place and
paint them in another. What is meant by reading and painting is the
reading of colour information of a point in the video, and the drawing
of this very colour on another point, keeping in mind only two rules:
it should not be painted in a completely arbitrary place; a random
walk would be used starting from the point where the original colour
was read, and that not all the pixels should be painted, only a small
amount. The only change from one state to the other lies in a slight
tendency of the random walk to take the points back to their original
position, where they constantly read their colour value. The fact that
not all of the pixels change allows for the drying effect to emerge,
which is nothing other than the traces of colour that a pixel leaves
behind when it returns to its original position.
So, they are simple rules that generate far more complex results than
might be expected from the mere analysis of rules, at least from an
aesthetic point of view. Which is why the concept of emergence was
mentioned earlier, as that is how the drying state came about. It
wasn't a planned or sought after effect, it came about when the highly
simple rules that underlie the piece's programming were carried out,
with small variations from the liquid state.
The result is a painting in a state of continuous movement that
imposes its rhythm on the viewer, or, rather, its surroundings. When
placed before an urban landscape, it turns into an interpretation of
it that reminds us of how painterly what is occurring before the
camera actually is. The installation was up for a few months in
Manhattan, pointing at the crossroads between Broadway and Waverly
Place. By night, the taxis' headlights would leave white and red
traces on yellow. The passersby, mostly without knowing, also added
some colour to the composition, and the newspaper stands would appear
and fade away continuously.
Within other environments, Liquid Video can be an intimate mirror that
plays with one's own image, slowly drawing out and blurring one's
features. Leaving traces behind and evoking multiple metaphors about
life and the passing of time.
4
Joan Soler-Adillon, Playing with Pixels

3. Painted
Painted was made after Liquid Video, and, in some way, can be
understood as an evolution of the latter towards something that is
actually an interactive installation. Painted was strongly inspired by
works such as Daniel Shiffman's Swarm and Rozin's mirrors.
Just like in the previous piece, this work is a digital painting that
tries to converse with its environment, but in this case it also
starts up a dialogue with the viewers; in fact, Painted only converses
with its environment when it detects movement - a symbol of life, or
at least of activity - within it. Similarly to the way a work of art
does not exist without eyes to look at it, this work only exists with
the presence and movement of the viewer's body. If nothing happens in
front of it, nothing within Painted changes. If something moves, a
section of the image will change so that it becomes impossible to come
close to the work without altering it. It is therefore a work that
constantly looks for change in order to respond to it. This is the
dialogue it offers. If nothing happens and nobody moves, the piece
also remains inactive.
This work, just like Liquid Video, uses pixels as its raw material and
video as a source of information. It also never updates a frame
completely. In Painted, only a small section is updated (changed) at a
time, based on a specific point. In fact, it uses the point, or one of
the points, where movement is detected, so that the viewer is also
subject to the rhythm of the piece.
Each section is repainted radially, from the interior to the exterior,
as if it were a centrifugal explosion of digital paint. Furthermore -
just like in Liquid Video -, in the redrawn area, not all the pixels
are updated, some are, some are not (once again, randomness is used).
This gives the painting a sort of pointillist appearance that makes it
more interesting.
The fact that only a single point can change the image at any given
moment also allows the user to only leave behind mere traces of his
presence. The video that is at the foundation of it all therefore
remains in the background after creating an almost painterly image.
Consequently, this is a piece that continuously looks for change and
movement, and only acts when it detects it. Just like Liquid Video,
Painted is a piece that demands the same space of a conventional
painting, but also offers interaction. Its ideal situation could be an
art gallery, but also the lobby of a busy space or even a street, as I
already described above.
5
Joan Soler-Adillon, Playing with Pixels

Painted
The piece has an aesthetic value, as its presentation offers a sort of
image that is much closer to the art of painting than that of
television. It is simply an invitation to look. But it also invites
participation and dialogue. If the viewer so desires, he can, on his
own or in a group, play with the making of the image. The rules are
simple enough to learn, or to sense, quickly, and the result is
elaborate enough for time to extend through an act of exploration,
which is in fact a participation in the very creation of the piece.
4. On Beauty
On Beauty is the third and last piece of this first series. It is a
proposal that asks for the viewer's participation in a more direct
way, and, in this sense, is a more conventional interactive
installation. Following on from previous works, On Beauty presents a
space with a white screen to the viewer, which is surrounded by the
subtle projection of an image that is being processed from a life feed
of a camera placed behind the user. Next to the screen, the viewer
will find
a paintbrush with a small built-in light. By moving this paintbrush
towards the screen, he will see how a still image unveils, following
his movements. This image is worked on through algorithms that are
similar to those used in Painted to present a real image in an almost
painterly in a sense, embellished, style. In this way, the viewer
finds himself in a technologically advanced environment (a plasma
screen. a projection. a modified paintbrush ...) and some images that
are, at first sight, just pretty images.
6
Joan Soler-Adillon, Playing with Pixels

The surprise lies in the content of the images, which is revealed by
moving the paintbrush. These contain images of children and adults
injured at war, and of victims of police abuse. Images that represent
social injustice in places that are far from advanced societies, where
the heavy presence of technology is often sustained by different
conditions in other parts of the world.
On Beauty
Hence the name of the piece: On Beauty is a modest claim, a small
outcry against the unquestioned acceptance of technology's pre-
eminence in advanced society, which often goes hand in hand with
opulence; the unnecessary and unbridled consumption of products that
have been fabricated under conditions that we would find, in theory,
unacceptable.
In fact, there was a clear element of provocation within the work, and
the question of whether it might be too risky a piece, whether it
might anger some viewers upon realising that they were painting
injured children, was raised. The fact is that throughout the two days
of the initial presentation, not a single user appeared to bat an
eyelid. Many didn't even realise what the image they were revealing
actually represented, perhaps because they were not looking for a
figurative image within it. Perhaps, also, we are already immune to
this sort of images.
On Beauty concluded this small trilogy, which was based on the
handling of real images. This third piece, which was different to the
first two works in terms of presentation, also opened the door to
proposing a greater implication from the viewer. All of this
influenced the fourth and final piece presented in this text.
7
Joan Soler-Adillon, Playing with Pixels

5. Digital Babylon
This story began with a professor, and I mentioned that it would
finish with another. This other professor is Daniel Shiffman, who I
had the privilege of being one of his first students. Thanks to his
enormous patience, he was able to untie the first knots of my
programming Ariadna strings, which were made up of functions and
variables, and made up some fabulous classes that inspired the last
project that is described here: Digital Babylon. Throughout the entire
process, he constituted the most valuable support.
Digital Babylon is an interactive installation that is very different
to the previous ones. In the first place, its aesthetic, which
abandons work with real images in order to concentrate on synthesised
ones. Synthetic and simplistic images, as the elements that make them
up are intentionally simple so that movements and interactions can
give the piece its intended feeling.
The piece presents the viewer with a small virtual ecosystem. The
system works without the need for the viewer's intervention, although
she can actually intervene in order to alter it. by entering a
designated space in front of the projection that presents a virtual
space. This installation was born out of the will to propose an
alternative to a process that was commonly found in interactive
installations. That is, one of a clear duality of states: a first
state of waiting, always identical, and a second of interaction.
Usually, once the second state is over, the piece simply returns to
the first state. Digital Babylon proposed to try and make an
installation that would accumulate each and every one of the
audience's interactions, to offer a proposal to the viewer that is
always changing. The point was to offer the viewer two levels of
interaction. A level where one finds and understands immediately, and
another, more subtle one, closer to the idea of a piece that converses
with its surroundings, that we might call cumulative interaction4.
The work, inspired by works such as A-Volve by Christa Sommerer and
Laurent Mignonneau, or Karl Sims' Evolved Virtual Creatures, offered a
double level of interactivity. One is obvious, with the viewer
entering the installation space, and another subtler one of
accumulation, developed through evolutionary programming. What follows
is a description of the work and its processes.
Upon entering the installation space, the user finds himself before a
projection of a black background. Within it, two types of visual
elements are moving and a third one gradually appears. They constitute
the two species that live within this virtual ecosystem with a third
element that they feed on.
8
Joan Soler-Adillon, Playing with Pixels

Digital Babylon's Virtual Ecosystem
The first species (which we shall call main species) is made up of
between a few and a few hundred of simple looking beings (small
triangles that change colour when their state changes). Its basic
functions are to feed and breed. If they don't do the first of these
enough, they die. And if they do it up to a certain level, they can
accumulate enough energy to attempt the second function. If the
attempt proves to be fruitful, a new being of the same species is born
and enters the chain. All of this is done whilst avoiding contact with
the other species, which is its predator, as much as possible. The
second species also has its own different ways of feeding and
breeding. It feeds at the expense of the main species, as it is by
eating them that it obtains its food.
The main species finds nourishment in plants, which appear where the
lifeless body of one of the members from either species lies, as if
its body were a fertiliser for the digital land. The predatory species
is aware of this, and always moves around the areas populated by
plants. If they find a member of the other species on their way, they
will attack it to trap it and eat it.
All these rules and interactions between each individual element
generate a considerably complex system, which in its own way creates
its own balance and rhythm. There are phases where there is a lack of
food, which lead to the death of many individuals, which in turn
generates a large number of plants, and, consequently, a new phase of
abundance.
Within this whole process, one may observe traces of the afore-
mentioned concept of emergence, as complexity sometimes appears to go
beyond the explanation that comes from the mere analysis of the parts
9
Joan Soler-Adillon, Playing with Pixels

that make up the work, especially if we consider one last aspect that
is yet to be explained.
Both species evolve through genetic algorithms. a computer technique
that was first described by John Holland in 1975 and then used by
artists linked to A-Life Art (art made by generating artificial life
systems). These works applied basic ideas of genetic evolution to
programming. In short: genetic algorithms usually begin with a
population of elements (agents, programming subroutines, etc.) made in
an arbitrary way within a number of predefined parameters (the
genotype): each of these elements (phenotypes) is then evaluated
according to predetermined criteria (fitness); the most successful
elements are selected to then create a new generation of the
population, which will be evaluated again, and so on. The variables
that make up each of the elements act as virtual DNA, which is re-
combined in consecutive generations. In this way, new individuals,
even if they are not the same as their progenitors, inherit the
characteristics that have made them successful. Finally, there is also
the possibility of applying mutations. That is to say, to randomly
alter one of the variables that conform some of the individuals. This
allows for the insertion of new possibilities into the system that are
different to predetermined ones, and which, if efficient within the
environment they emerge in, will enter the evolutionary process.
All of this is applied to the two species of Digital Babylon. Every
time two individuals of the main species mate, a new individual
appears, inheriting their abilities. Here, as in other A-Life Art
pieces, there isn’t a predetermined fitness criteria, Their success or
failure depends on their interactions with the rest of the elements
within the virtual ecosystem and, as we shall see, on the viewers'
actions.
With regards to the predatory species, the process is very similar.
Although the reproduction process depends on its actions as a group,
the result is the same: only individuals who obtain a certain amount
of success within the environment will be able to reproduce.
All of this makes up a complex and, at the same time, changing
environment. Both species are constantly evolving. When only a certain
type of individuals reproduce, the species as a whole gradually
changes, and this in turn affects the rival species5. If we finally
add the audience's actions to all this, we may come to an
understanding of the idea of a double level of interaction. The way
this interaction is produced is, simply, through the user's presence
within the installation.
There is no other interface apart from the body of the viewer. His
presence within a determined space in front of the screen leads to the
appearance of a small dot that represents him within the virtual
ecosystem. When this happens, and as long as they are not too busy
eating and trying to mate, each of the individuals of the main species
will tend to come more or less close to the newcomer, depending on
10
Joan Soler-Adillon, Playing with Pixels

their virtual DNA code. So, some individuals will completely ignore
him. and others will follow him everywhere. From this moment onwards,
the viewer can decide how to interact with the individuals that follow
him. He may decide to help them, or might do the opposite, which means
he will put his followers in danger of being eaten by the predators.
A Visitor in Digital Babylon
Depending on what he does, he will make the individuals who are more
prone to come close to visitors, or the more unsociable ones, have
more or less possibilities of survival (and, therefore, of
reproduction and of passing on their characteristics to consecutive
generations). He will affect the interactions of future users of the
work, making it easier or more difficult to interact with the
individuals of this first species, depending on the sum of all the
users' interactions.
It is in this sense that we can talk about a double level of
interactivity: first, there is immediate interaction, where elements
react to the visitor's presence; secondly, there is cumulative
interaction, which is made up by each and every one of the users, and
which makes the piece change a little with everyone, without any of
the subsequent interactions having the capacity to eliminate the
effects of those which were produced before it.
This double interaction, together with constant change due to genetic
algorithms, is Digital Babylon's main proposition. The point is for
the work to modify each of the individuals' movements; how members of
the same species interact, how they interact with those from other
species, with the environment, and how those of the main species
interact with the user of the piece, as a result of all the different
interactions and of evolutionary programming. The intention is that in
11
Joan Soler-Adillon, Playing with Pixels

this way, if the installation is active during a long period of time
(various days or weeks), the changes might be substantial enough, for
a viewer who might return to perceive the work differently, and find
it interesting once again.
Coming Full Circle
Five years after the process described above, looking back on it with
heavier baggage, has been an interesting experience. When one revises
the ideas of a not so distant past, one sometimes sees new connections
and new points of interest in them. From the distance a dilated,
active, time lapse provides, some of the lights that I used to see in
the pieces whilst making them have certainly diminished, but others
are still active in the same way, or with stronger conviction. Other
lights now appear to be new, new points of interest such as the
connection between the first and last works, described in terms of
processes of emergence. Digital Babylon remains a valid proposal. It
is still valid to look for new levels of interactivity within art, to
involve the viewer in subtle ways, to turn him into an accomplice of
imbalance. And it is also still worth looking for a dialogue between
the works and the environment, the way Jim Campbell demanded in his
2000 text.
He called for an interactive art that would converse with its
environment, which had to be done through works that avoided constant
repetition. The paragraph with which he finished the text that
presented this idea, in spite of being a decade old (an eternity in
terms of digital art), retains its strength, from its first assertion
up until its final parenthesis: "The possibilities exist for works
that perceivably never repeat themselves. Works that respond to their
environment not just in a short-term way, but in a long-term way,
unpredictably and meaningfully (easier said than done)6."
12
Joan Soler-Adillon, Playing with Pixels

1 The published text in the book contains errors in the chapter sections and also in Jim
Campbell’s final citation, which have been corrected here along with some details of
the English text. The images in this PDF version are not included in the book.
2 For a general explanation of the concept of emergence applied to the digital arts, and
to consult a list of references, see Citation: Soler-Adillon, Joan (2010). "Emergence
and interactivity: A-Life Art as a paradigm for the creation of experiences in
interactive communication". Hipertext.net, 8,
http://www.upf.edu/hipertextnet/en/numero-8/a-life_art.html. See also Soler-Adillon,
Joan (2011). “Creating Black Boxes: Emergence in Interactive Art”. ISEA 2011
Proceedings: http://isea2011.sabanciuniv.edu/paper/creating-black-boxes-emergence-
interactive-art
3 The artist Jim Campbell proposed the idea of a dialogue between interactive work and
its environment in an article from the year 2.000: Campbell, J. (2000). Delusions of
Dialogue: Control and Choice in Interactive art. Leonardo, 33, 133—136.
4 The notion of cumulative interaction is described in more detail in the first article
mentioned in note 2.
5 The results of these processes can be very surprising. During a particular stage of
the process, the Digital Babylon prototype was left working for three hours in a row.
After this time had elapsed, whilst in a group, the main species moved and (apparently)
acted in a way that was surprising, even for the actual programmer (video documentation
of the results can be seen online: www.joan.cat/project.php?id=1).
6 Idem note 3.