Content uploaded by Ernesto Peña
Author content
All content in this area was uploaded by Ernesto Peña on Nov 27, 2016
Content may be subject to copyright.
Journal of the Canadian Association for Curriculum Studies (JCACS) 108
Volume 14, Number 1, 2016
A Glitch Pedagogy:
Exquisite Error and the Appeal of the Accidental
Ernesto Peña
Kedrick James
&
the Digital Literacy Centre
1
Department of Language and Literacy Education
University of British Columbia
Abstract:
By experimenting with computer glitches as provocation of accepted norms of user
interactions with digital technologies, this paper extends and radicalizes Dewey’s (1934)
pedagogical principle of “consummatory experience,” observing computational error,
logical accidents, and procedural glitches as creative and productive forces in the lived
curriculum. We hold that this troubling of expected outcomes, this disruption of
programmed processes which, as a result of incommensurable informational input,
result in unique (and educational) by-products, is fundamental to understanding our
digital humanity, and that these irregularities convey the same learning potential that
learning from mistakes and fortunate accidents do in the arts, sciences, and within the
broader context of lifelong learning.
Keywords: glitch, creative pedagogy, automation, machine learning, John Dewey,
consummatory experience
Glitch Pedagogy
JCACS
109
Introduction: Accident and Error
t is not difficult to see the overwhelmingly negative stigma that is attached to accident
and error; the irony of appreciating such mishaps seems almost comical. We are
taught from an early age to experience embarrassment and humiliation when we
make mistakes, but good teachers know how to work with our mistakes and activate
our learning by engaging with those results rather than inculcating our fear of errors. The
commonplace myth that children learn languages easier than adults arises from the freedom
that very young children feel to make errors in speech. Adults indulge and might try to
understand a child, where a fellow adult is judged accordingly. So we internalize the
reluctance to try new things as we grow older, afraid of not being good at something. All
learning relies on the ability to try, within a communicative context, to comprehend and
express oneself strategically, whether through a golf swing, growing a garden, making a
Vine video, or reading the spectrogram of a star. Error is at the base of logic, as logic is the
attempt to reduce and control forms of error, and here logic is pattern recognition on the
part of an observer. The keener the background knowledge of the observer the more
precisely error can be identified. When processing any kind of familiar pattern of input data,
whether organically through the senses, mechanically, or digitally, error alters the rate of
activity of the processor. It does not slow it down; it greatly speeds it up. A computer hard
drive running error messages starts to overheat.
Errors can cause the entire system to fail. We might wonder how something so
seemingly insignificant as a missed quotation or colon or missed DNA pairing can have such
dramatic outcomes. That is because all controlled or coded events (which is all creative
phenomena) entail both a goal, sometimes quite a simple goal, and a very complex series of
constraints that are like gatekeepers to achieving that goal. Those programs running
differential software such as Bayesian and Markovian algorithms make highly proficient
(indeed intelligent) pattern recognition machines. As such, they work on calculating
probabilities from a wide range of known and unknown variables, trained on achieving a
particular goal. Key here is the notion of probabilities, and that probabilities are guestimates
within a sea of error. Stuart Armstrong (2015), for example, explored the difficulty of
training artificial agents to make decisions based on values instead of facts. With utility
(ease, speed, efficiency) as a coefficient, “constructing a well behaved value selecting agent
immune to motivated value selection—one that is capable of learning new values while still
acting on its old ones, without interference between these two aspects – is an important
unsolved problem” (p. 19). Using the Sophisticated Cake or Death problem, a logistical
challenge, results in a lot of Death. The point here is that even artificial agents need to learn
from their errors to make good decisions.
Errors in the age of industrial mechanization lead to bad accidents, but other than
human miscalculation, errors were initiated by irregular input in automated manufacturing.
If the goal was to produce a certain type of lamp, for example, it was possible to mass-
produce all the intricate parts in a manner that removed humans from the process almost
entirely. One type of machine could be made that produces filaments, another bulbs,
another metal housings, another wooden shafts, another insulated wire, another plugs,
while another cuts and glues felt shades and so on. Once each job is standardized,
sequenced, and mechanized; mechanical automation promised to eradicate those pernicious
faults of human error. However, humans, even at the height of industrial mechanization,
remained vital to manufacturing to solve the conflicts of non-standard materials to be fed
into the machines. Where a carpenter, seeing an unusual warp in wood around a knot might
decide to work with the knot and bring out its irregular beauty, the same piece of wood
going into a joinery machine could damage or destroy the machine. In contrast, it has never
I
Peña & James
JCACS
110
been possible to fully automate the shoe industry. Lamps could be mass-produced because
from year to year, the materials remain relatively constant. If however, owing to the
changing designs and materials of the fashion industry, every year you needed a new
machine that could work with leather to produce a stiletto, then another to make boots with
snakeskin, then another to make sneakers with textiles, then another to make gumboots
with rubber, then another to make flip flops with plastic, the cost of automation would never
be recouped (Bright, 1958). Moreover, your company would always be one year behind the
current market, given that by the time the machinery was produced, the trend would have
moved on.
All this teaches us about the role of error and accidental input from a technical point
of view. However, it is not from industry that our understanding of error and accidental
beauty arises, but from art. During the era of the rise of industrial automation and
concomitant human vs. mechanical agent conflict (Noble, 1986), America was also in the
golden age of jazz music. What spotlighted jazz as particularly significant for music
enthusiasts was the way top players of the genre would embrace improvisation (Menezes,
2010). Often the improvisor would feed off other players and/or members of the audience.
While they retained a common goal, they also embraced the possibility of irregular input
and unusual results. Supporting findings by Pressing (1988), Menezes describes
improvisation “as a skilled performance with error-correction capabilities (a closed-loop
feedback system) coming from the real-time comparison between intended and actual
output” (p. 11). Players would adapt and support each innovative gesture, expanding
musical possibilities into expressive probabilities. Improvising musicians “view errors as a
motor of creativity” (p. 57). Music is by no means the only art form that employs error to
accelerate creativity.
Improvisation is also central to explorations of glitch. Because this research focuses
on provoking unpredictable outcomes from routine processes, it requires invention,
imagination, and improvisation within a series of imposed technical constraints.
Improvisation is a form of inquiry with skill. When still at basic stages of learning routines,
errors can be corrected according to a known set of goals and parameters. When moving
past the basic goal of acquiring patterned expression to a new goal (e.g., expand the
listeners’ aesthetic parameters, both known and unknown), then errors are more difficult to
distinguish from exceptional results; errors initiate new learning, inspire adaptation, are
skilfully integrated as a new fundament, and used to demonstrate mastery of the (art) form.
Hence, with basic learning complete, improvisation as a methodological technique enhances
speed and dexterity by which the artist incorporates error-as-learning cycles and turns
mistakes into a form of innovation and beauty.
This process of transforming error into learning and then transmuting it into
metaknowledge is at the base of what John Dewey anticipates with the concept of
consummatory experience, which he theorizes in Art and Education (1934). According to
Dewey, engaged inquiry in the processes of achieving an outcome is as significant, or more
significant, to learning than the product that is produced at the end. Likewise, Dewey’s
insistence on processes of learning being emphasized over product was a central tenet of
the Process Writing movement, which sought to introduce to the teaching of writing the
notion that multiple states and processes were typically undertaken when any given piece of
writing was composed (Elbow, 1981; Emig, 1971). The goal, obviously, was to write
something in a particular style and genre, but the constraints were many, some known
(formal genre properties) and others unknown (audience reception, even exhaustive
knowledge of the topic). Formal experimentation with processes could lead to better writing
as the writer learns by experimenting with the compositional process.
Glitch Pedagogy
JCACS
111
Inquiry and Technology
Dewey understood inquiry as an unavoidable, pervasive human activity: “[inquiries]
enter into every area of life and into every aspect of every area. In everyday living, men
examine; they turn things over intellectually; they infer and judge as naturally as they reap
and sow, produce and exchange commodities” (Hickman & Alexander, 2009, p. 170).
Dewey’s notion of inquiry has been qualified as democratising, a breakout from the
dominance of theory over practice that classical philosophy contributed to disseminating and
that has ruled the mindset of western civilization (e.g., Schön, 1992). For Dewey, inquiry is
not a merely intellectual process but a hands-on approach to questioning in which the value
of tools of thought (theories, concepts, constructs, etc.) does not rest on their accuracy, but
in their ability to convey further knowledge; that is, knowledge obtained from inquiry is a
tool for inquiry itself (Chunn, 2014). The emergence of digital technology and its
increasingly protagonistic role in education has sparked several new discussions, but it has
also opened the possibility of re-examining others that could have been considered settled
from a new perspective. Dewey’s thought has not been the exception. The space that digital
technology would allegedly take in Dewey’s theory of inquiry has been discussed since at
least the early nineties (i.e., Hickman, 1990), although more recent accounts have
problematized this role by bringing another Deweyan concept to the discussion, a
pedagogical principle known as consummatory experience. According to Dewey (1934) a
consummatory experience is understood as the interplay of the means and the ends as a
condition towards a culminating, valuable experience, even if that experience is not
necessarily fulfilling or pleasurable. The coinage of this concept also reflects the critical
stance that Dewey had on the culture of his time for alienating processes from products
(Tiles, 1990). These consummatory experiences are the kind of experiences that should
emerge as a direct outcome from teaching (Oral, 2013). These two concepts: inquiry and
consummatory experience are inextricably connected, as it is allegedly through reflexive
inquiry that an experience can ultimately be consummatory in Dewey’s terms.
Scholars like Eric Mullis (2009) have denounced the use of proprietary digital devices
and applications for educational purposes because such use puts the concepts of
instrumentalism and consummatory experience at odds. Even when these devices can be
used as effective tools for inquiry, their proprietary nature is oriented to the fulfillment of its
intended purpose by hindering the underlying processes, and by doing so, circumstantially
preventing a consummatory experience by blurring the means for inquiry. In other words,
by dealing with the whats without addressing the hows. This statement might be debatable
in cases in which there is no analogue version of an activity (computer language coding, for
instance) that a digital device is emulating. The use of these devices and proprietary
applications has been promoted in western education models as a practice towards the
acquisition of digital literacy (e.g., Johnson, 2011; Leonard, 2013; TeachThought Staff,
2014) to the degree of having institutions holding conferences on the use of certain mobile
devices for educational purposes (e.g., http://www.tlipad.com/). The criticism and
drawbacks (Murphy, 2014) of using proprietary technologies primarily focuses on economic
and logistical challenges of the extensive implementation of such devices and not on the
pedagogical outcomes of their use. This potentially occlusive aspect of technology was
foreseen by other influential voices in education and pedagogy such as Ted Aoki. Back in the
late eighties, Aoki (1999) warned about the dangers of an overconfidence in computer
technology insofar as it only accounts for a limited standing reserve of manifestations of the
real, instead of revealing other possibilities:
we must seek the true by understanding computer technology not merely as means
but also as a way of revealing. As a mode of revealing, computer technology will
come to presence where revealing and unconcealment can happen, i.e., where truth
can happen....Hence, what endangers man, where revealing as ordering holds sway,
Peña & James
JCACS
112
is his inability to present other possibilities of revealing. In this, it is not computer
technology that is dangerous; it is the essence of computer technology that is
dangerous. (pp. 170–171)
Here, we do not try to engage in a discussion about the pedagogical value of proprietary
digital technology or computer technology in general; instead, we contend that digital
devices can be used as teaching and inquiry tools while preserving the consummatory
experience and the potential to reveal alternative realities to some extent by adopting the
concept of glitch as a prompt for inquiry.
Glitch
According to the Oxford English Dictionary (OED), the noun glitch started to be used
among astronauts to refer to a momentary electrical spike over which there is no control.
An early use of this word in this context reads: “A glitch is a minute change of voltage in a
wire which is enough to trigger another system out of proper sequence” (Stimson Jr., 1961,
p. 232). This notion of lack of control can be seen as well in the verb to glitch, originally
defined broadly as to “meet unexpected problems” (“glitch, v.,” n.d.). Since the early sixties,
the term has evolved to encompass unforeseen behaviours within a system, particularly—
although not exclusively (e.g., Willis, 2007)—to computer systems, and has been mobilized
to other fields, such as video gaming (e.g., Krapp, 2011) and art (e.g., Menkman, 2011)
among the most prominent. In the realm of computers, the rhetoric around glitches inclines
towards an understanding of undesirable occurrences and errors, where glitch is sometimes
used as a euphemism for bug (e.g., Krutz & Lutz, 2013). However, unlike its understanding
in relation to computers, in gaming and art-related popular media, a glitch is seen as a
serendipitous event, privileging unexpectedness over failure (e.g., Hernández, 2012).
Moreover, in the specific case of glitch art, it has been argued that the lack of control is,
more than a circumstance, a desirable ontological condition (e.g., Menkman, 2011). A glitch
can only be so if there is no control over the result. These dispositions towards the
phenomenon of glitch resonate with us and our proposal in regard to proprietary digital
technology, although we are aware that precisely because of the proprietary nature of many
devices and applications, the room for glitching is limited to the affordances of the tools and
to the previous experience of the witness of the glitch. In other words, a glitch requires a
norm to digress from, and it is possible to assume that a glitch could be understood,
predicted, and ultimately become a norm. It is also possible to assume that what is a glitch
for one person might be a norm for another, depending on their previous knowledge. It is
because of this that in our own understanding, a glitch is a non-moralizable, contextual
event.
Although we acknowledge the dual character of glitch as a product and a process,
our approach to glitch is that of a product instrumental to the understanding of its own
process, a sort of reverse engineering in which the goal is not to reproduce such processes
but to shed enough light on them to make them tools of inquiry. We would argue that
glitches can serve to interrogate not only the underlying processes that lead to the glitch
object, but would also disrupt the standing reserve of realities that Aoki denounced (1999).
Moreover, glitches could help to interrogate the very definition of collaboration, agency,
authorship, intentionality, and so on. Consider for instance the following questions: Who is
the author of the glitch as digital artifact? Is it the user of the proprietary device when
producing the glitch? The programmer of the algorithm that the user intends to disrupt? Is it
the device itself? All of the above?
Glitch Pedagogy
JCACS
113
Method: Cases, Experiments, Protocols, and Provocations
Although the concept of glitch as we present it here could be understood as a mere
theoretical tool, we envision it as a hands-on form of inquiry. Glitch is not only something
that should be thought about or observed. Glitch is something to be provoked, even
performed, and as such, glitch is also provocative of methodological innovations.
In order to illustrate the arguments here exposed, we present four cases of the use
of glitch as a tool for inquiry. The first three cases deal with different technologies, methods,
and formats, respectively static imagery, automated literacy, and language translation
protocols. The fourth case applies a recombinant process that utilizes the above three
methods to produce a distinctive artifact that has been glitched at each stage of production.
These cases should be taken as mere examples of the potential of the adoption of glitches—
unexpected products of purposeful disruption—as pedagogical and reflexive tools. It is
important to point out that all these resources are available in most of the current devices
and operating systems. These cases are: provoked panoramic camera mistakes (referred to
here as stitch-skipping), automated voice-to-text fuzzy interpretation (voice-vaguing), and
natural language machine translation errors (glot-swapping). These particular kinds of
glitches are based on either open or native applications commonly available across
computer platforms.
Case 1: Stitch-Skipping
Typically, artifacts designed for specific purposes provide—or should provide—
information about their use. Possibilities for action or affordances are often related to
physical objects and are ideally inferable from even very basic designs (Bransford, Shaw, &
Minnesota, 1977; Norman, 1988). The notion of noticeable hints of the use or interaction
possibilities of an artifact is referred to as perceived affordances (Gaver, 1991). For instance,
by virtue of their shape, the affordances of objects such as hand axes, hats or spoons
should be clear enough to require no further explanation. The concept of affordance has
been also applied to digital interfaces, usually mediated by other semiotic resources.
Sometimes this mediation is done subtly through interface metaphors, for example, where
the affordances of artifacts familiar to the users are insinuated in the interface. Sometimes
this mediation is as obvious as with skeuomorphic design, where the general aspect of an
object is mimicked in the interface. In these information environments, explicitness of
affordances makes intuitiveness a highly desirable feature. It can also occur that the
affordances of an object or an interface are not perceivable but hidden (Gaver, 1991). In
those cases the operation of the artifact would require explicit instruction, unless the
function of artifact is to make the user discover those affordances, as with some video
games.
Glitches can be provoked by acknowledging and willingly disrupting the perceived
affordances of an object or system, or by receiving explicit instruction on how to interact
with such an object or system and going against such instruction. Stitch-skipping consists in
glitching panorama pictures by purposefully digressing from the protocol for their creation.
These images display an elongated view of the selected scene that would be impossible to
capture in an analogue photographic format. The process of creation of these kinds of
images varies between devices and systems. However, it is possible to assert that, in
general, in mobile devices–at least in the two currently leading operating systems
(International Data Corporation, 2014)—the process of creation of these kinds of images
requires the user to perform a relatively stable sweeping movement across the scene using
the device’s camera software. Mobile devices with panoramic picture capabilities usually
Peña & James
JCACS
114
provide basic instructions or indications of the procedure for production of a canonical
panorama. Although it could be assumed that the available ways to disrupt the standard
procedure for the production of panoramas vary on each system, these procedures of
production would be the first aspect to be disrupted due to their accessibility, as the
pervasiveness of both intentional and accidental panorama glitches in social media seems to
prove (“Get Glitchy with Your Phone’s Panorama Function,” n.d.; Lowe, n.d.; Neff, n.d.), to
the extent that it is relatively easy to find online tutorials for glitching panoramas. In these
tutorials, the expected result of a photographic panorama is disrupted by the movement
either of the camera or of any of the elements in the scene while the picture is taken. This
implies that the norm for the creation of panoramas requires objects and subjects within the
scene to remain still during the capture. Failing to comply with this condition, disregarding
the systemic requirements of the software employed, results in a glitch.
The starting point of our inquiry was an attempt to understand the role of movement
in the occurrence of glitches in panoramic photos by purposefully moving the camera, the
subjects, and objects in non-normative ways. The first step consisted in creating a reference
image in compliance with the norm or the expected outcome of a panoramic photo. After
that, using conditions as similar as possible to those of the first shot, subsequent images
were
in which either the camera or the elements in the scene were intentionally moved.
The result was a normative image and a set of glitched versions of it. The first image was
then compared with the set of glitched images to observe the differences and devise
possible explanations.
Glitch Pedagogy
JCACS
115
Samples 1 & 2 (Top to Bottom):
Canonical Panorama; Moving Subject; Horizontal Motion; Vertical Motion.
Panoramic images break the ergonomic conventions that many optical devices such
as cameras or displays have. These conventions (landscape orientation with a given
proportion, either 4:3, 16:9, etc.) are arguably based on the distribution of the human eyes
in the face (Skopec, 2004). Panoramic pictures extend the perception of a person beyond
what can be perceived within a normal gaze and by doing so, alter the common narrative
capabilities that regular photographs have (Nelson, 2007). Panoramas are created by
assembling a sequence of pictures based on the similarity between the edges of the
contiguous images (a process known as stitching). Considering the infinite number of
factors and parameters that would play a role in determining when two images are
contiguous, the stitching program allows for a generous margin of error, and it is within this
margin of error that glitches emerge. Typical of glitches found on social media, the glitching
agent is a person or an animal that moved during the take. In these images, the glitch
reveals this movement, and by doing so, it reveals that panorama pictures are constructed
from a sequence of discrete images, meaning that, in particular conditions, a glitch could
denote time and sequence—features alien to standard photography. In addition to working
with movement within the frame, we also experimented with provoking a glitched image by
irregular movement of the camera while shooting. By choosing to not comply with the
steady sweeping motion, the stitching process of the algorithm is swamped with incoherent
data. The resulting product ends up being a visualization of the camera movement, as long
as it follows a particular course; in other words, as long as the camera’s motion has a
modicum of coherence. In both cases, the resulting glitches are visualizations of the
sequential processes that went into their making.
Peña & James
JCACS
116
By doing so, the taker of the photo is formally represented in the resultant image,
even though they do not appear as subject. This makes the malleability of representation
much more obvious to the observer encountering a glitched image that captures something
of the intention of the photographer. Typically, such representations remain indexical,
pointing toward an assumed external reality from which the photographer, as the symbolic
agent, remains absent. The invisibility of the photographer, the media, or the distribution
network, hampers the learner’s ability to develop and utilize critical thinking skills in regard
to the politics of representation. This visual glitching process disrupts the illusion of reality
conveyed by representational media, enhancing sensitivity to the processes underlying the
creation of narratives of truth and fact.
Case 2: Voice-Vaguing
Building on work that began in the 1930s (Huang, Acero, & Hon, 2001; Juang &
Rabiner, 2006), the outstanding achievement of automated speech recognition or speech to
text (STT) software symbolizes the ability to program machines to overcome challenges
similar to those that “humans face in understanding language: linguistic analysis of input
(deciding what was actually said); semantic processing of the input (interpreting what the
input means); pragmatic processing of the input (decisions on how to respond to the input)”
(Rost, 2011, p. 99). In order to achieve these goals with a level of dialogic fluency,
computers require both substantial background knowledge (training) and robust
computational tolerance for variations of acoustic input, the latter requiring far greater
memory and processing capacity than is required for processing print text which benefits
from discrete orthographic tokens and units. Speech, on the other hand, is continuous and
does not distinguish homophones (e.g. there, their, they’re) except through context.
Automatic processing of natural language is “now used for a wide range of applications such
as information extraction, machine translation, automatic summarization, and interactive
dialogue systems” (p. 99). Recent advances in mutual processing by machine-learning
algorithms have culminated in the development of neural net and deep learning models of
speech processing programs that have helped speech recognition become a standard
human-computer interface option. This development alerts us to the fundamental changes
that will be brought about in many fields of education, and in particular to language and
literacy education.
Deep learning algorithms are capable of successful word recognition across a broad
spectrum of voices in many languages and dialects, and, moreover, are not voice-
dependent in their training (Larson & Jones, 2012). Gone is the comparison of each vocal
sound in real time to a standardized linguistic unit. Deep learning algorithms consider larger
clusters of speech-sound, predicting the expression by spectral analysis based on a deep
knowledge of underlying oral language structures currently in use by referencing big data
rather than matching frames of utterance (Rost, 2011) to sounds made by an idealized
speaker. In other words, these algorithmic systems have learned how to listen, understand,
and in the case of artificial voices, to speak fluently in a so-called natural language through
deep listening to billions of human utterances; they are deep learning in so far as they
aggregate big data to increase tolerance of variation and accuracy of transcription, and have
the ability to retrieve information that originates in human to human conversation as
computer data based on their programmatic instructions. Since the nineties, for example,
the U.S. National Security Agency has used voice recognition to scan phone and audiovisual
networks for keywords, non-normative communications, and voice-prints of persons of
interest (Froomkin, 2015); conversely these technologies are used in modern language
education to assist humans to acquire and correct speech. By the nineties, commercially
viable speech recognition software had reached a public with smaller average vocabularies
Glitch Pedagogy
JCACS
117
than that recognized by the software. This discrepancy between individual human speech
pattern recognition and that of deep learning algorithms has grown exponentially since 2010
(Deng, et al., 2013), to the point where it is even possible to scan transmissions of distorted
or barely discernable conversation.
Development of natural language processing software is so rapid that if David Pogue
(2010) was correct and “the keyboard isn’t going away in our lifetime...99.9 percent
accuracy is darned good—but until it reaches 100, speech-recognition technology [will] still
[be] plan B” (p. 40), his prediction will have more to do with the intractability of habit than
with the capacity of intelligent algorithms. Nonetheless, errors in interpretation abound even
with sophisticated software, and without normative pronunciation or regular syntax,
algorithms can no longer accurately predict utterances and are slower to compose. Our
intent is to enhance understanding of current limits of deep learning speech recognition
systems using voice-vaguing to exploit creative and pedagogical potential by increasing the
ambiguity of language being input into the processor. The processor analyses incoming
speech by filtering the waveform in layers, feeding in slowly adapting knowledge of linguistic
patterns in the target language, and making predictions based on previous words spoken
and current words to anticipate what the speaker will say next. This is a multi-tasking, feed-
forward system of analysis that greatly speeds up the process of comprehension and
emulates human linguistic processing, as we often guess another speaker, putting words in
their mouth. This feature of anticipating linguistic expression is now standard in mobile
phone technologies where words are both suggested in the process of being typed and
changed instead of corrected; popular social media sites like www.damnyouautocorrect.com
feature particularly funny or egregious predictive mistakes. Speech recognition failures are
now also becoming popular entertainment as well. It is not necessary to understand the
technical details of language recognition software in order to utilize these tools in a manner
that is both entertaining and alerts learners to the broader implications of their use of
mobile information and communication technologies.
We began researching speech recognition from the perspective of glitch pedagogy,
by using improvised and extended vocal technique to create ambiguity and il-logical
interference in the input data (voice-vaguing). This involved several stages in the
generation of a textual artifact, one in which both human and artificial voices were used to
vary input to the speech recognition software. Although proprietary, use of this software
enabled access to deep learning algorithmic responses. Following an iterative remix process
useful in developing critical media literacies (James, 2015), a selection of online videos
featuring infants babbling (in their sleep, in conversation as twins, and playing on the phone)
were input for voice recognition. Age extremes are an admitted weak spot in voice
recognition software (see http://atmac.org/speech-to-text-dictation-software-for-os-x),
which struggles with young voices (due to distribution of formants and fundamental
resonances in the waveform). Because voice recognition software uses spectral analysis
which filters out midrange tones and correlates low frequency resonances with co-efficient
high pitches, one is relatively assured, with infant dictation to voice recognition software, of
creating glitches. Regardless of the cultural group or language background of the infant,
voice recognition reduced complex babble to one word, mainly “are,” “or,” or “her”. What is
particularly startling about these results from the humanist perspective is that even though
the infants’ babble sounds like speech with inflections and pauses modeled on adult
language, the results are typically monosyllabic. For the example given in Sample 3, the
introduction of other linguistic items primarily results from noise or adult voices in the
background. Results for children up to the age of five continued this trend with very little
accuracy even when it is relatively easy for a human adult to decipher the child’s meaning.
The baby-babble text resulting from this experiment became our data source for further
iterative glitch experiments.
Peña & James
JCACS
118
Sample 3. Baby Babble Text.
Case 3: Glot-swapping
Glot-swapping trades the vocal sounds of one language with another, such as
expecting English diction to be captioned in Swahili. Typical uses of auto-captioning
software include subtitles for persons with deafness or environments that require the sound
on televisions to be muted. For our purposes, auto-captioning was activated in English to
caption video versions of non-English children’s stories. The software combines both speech
recognition and language translation algorithms which multitask as learning entities
embedded within deep neural networks (Yu & Deng, 2015), just as someone learning a
second language must do. As a glitch pedagogy experiment, we misrepresented input,
challenging the translation/captioning process, whereby the software must homophonically
translate the spoken words into an English equivalent.
Compared to the babble text, with adult voices the language classifier had more
success at producing a diversified textual output, formulating probable English utterances
from Russian, Spanish, and Hungarian source material. What is most remarkable about
these examples of homophonic auto-translation is the frequency of words pertaining to
corporate entities and national and religious identity (see Sample 4, Ёжик в тумане
(Hedgehog in the Fog, Norshteyn, 1975) closed-captioning of
https://www.youtube.com/watch?v=_Rugwd8ZNHY). A critical lens on the concomitant
texts might focus on the dominant presence of corporations such as Facebook and Google in
Big Data—perhaps a value-motivated skewing of the artificial agents—and inspect the
nationalistic and imperialistic overtones of the probability-generated phrases, which are
Glitch Pedagogy
JCACS
119
made that much more perverse as mis-readings, or glitch-readings, of children’s stories.
Seeing the bigger picture of language-use allows the glitch pedagogue to extend their
lesson-based experiments with captioning software to examine how our computerized
interlocutors can make errors and how these can be inherently biased, which potentially
results in unwarranted accusation or suspicion.
From a critical literacy perspective we would emphasize that while the algorithms
learn from us, we are learning from the algorithms. In fact, from our brief experiments, one
might say that such auto-captioning software drawing on big data as stored on United
States-based servers is a tad xenophobic. Xenophobia in algorithms, or at least a sense of
cultural bias in the programming of such algorithms, has been explored by researchers
working on automated language assessment for international literacy testing items and
evaluation protocols (Maddox, 2015) and in the analysis of Google’s technologies and
algorithms (Noble & Roberts, 2016). Recent glitch-tweeting escapades of Microsoft’s racist,
drug-smoking, teenage-girl, Artificial Intelligence chat-bot garnered unwanted international
attention for adopting these very human traits (Gibbs, 2016). One does not think of
artificially intelligent systems, robots, or international testing items as having cultural or
racial biases, but there are clearly indications that they represent biases inherent to the
discourses of dominant user groups. Once again, there is room here for critical literacy
scholars to take a much closer look at these instrumentalised paradigms of automated
language processing as governments and large institutions increasingly rely on computer
generated, artificially intelligent interpreters.
Sample 4. Ёжик в тумане (Hedgehog in the Fog). Closed captioning.
Peña & James
JCACS
120
Case 4: Toward Iterative Glitch
Along with glitch artists, we posit a perverse desire to witness deformity in the
becoming of digital objects, so that we might see this object anew and awaken our senses
to the appeal of the accidental, the beauty of errors in processing. In the same way that a
child amazes the adult with uncanny yet poetic and wise utterances, we seek to be
enlightened and entertained by unexpected outcomes. We fast-track our learning and
engage at the level of consummatory experience with the otherwise mundane or routine
software applications implicated in daily literacy practices. Continuing in this autodidactic
glitch pedagogy, we undertook the production of texts that would combine all the previous
elements—voice-vaguing, stitch-skipping, glot-swapping, using text-to-speech artificial
voices to read multilingual texts to voice recognition systems, a code-bending practice
which brings Optical Character Recognition (OCR) and Google’s translation software into the
recombinant error-prone process of iterative glitch-text generation. Owing to the
unpredictable outcomes of the increasingly complex tasks set for probabilistic calculation,
we are repositioned as learners, even while teaching with glitch pedagogies.
For this recombinant case in which an iterative method of provoking errors in
human-computer semiosis was employed, the text was generated with speech recognition
software trying to recognize the vocalizing babble of children, from infants to five-year-olds.
The resultant script (see Sample 3) was printed out on standard letter format white paper,
and a glitched photo panorama was taken using a mobile phone. Saved as an image file, the
graphic text (which resembled concrete poetry, see Sample 5) was uploaded to a free online
OCR program. The OCR recognized the distorted and recombined graphemes as letters and
ideograms from a broad cross-section of languages. It was enjoyable from a pedagogical
point of view, to have languages specialists in our academic department deciphering this
multilingual polyglot text (see Sample 6). Running the computer’s text-to-speech program
identifies the unicode character before speaking it, which gave us a way to true our
predictions, turning this process into a highly educational glot-swapping game.
Sample 5. A Glitched Panorama Photograph of Sample 3.
Glitch Pedagogy
JCACS
121
Sample 6. OCR Interpretation of Sample 5.
In addition, an improvised musical recording of a live vocal performance of the baby
babble source text accompanied by amplified analogue and synthesized soundscape was
produced. This audio file was also played to a computer running speech recognition software
so that a comparison could be made between textual products of analogue and digital glitch
experiments. In comparison, the computer-generated textual remix of the children’s voices
is highly complex and sophisticated. Stitch-skipping produces distortion at a graphemic level.
By comparison, the noise distortion factor of voice-vaguing with musical interference in data
for speech processing of spoken words results in a more visceral text with corporate/political
overtones, referencing Google several times, and current U.S. President Barack Obama
(neither of which is present in the song form, see Sample 7). These results correspond with
similar observations made in the Case 3 glot-swapping experiment. This comparison of
results between graphically glitched and sonically glitched source texts provokes further
stages for research needed to explore the potential for transmediation across sensory
domains.
Peña & James
JCACS
122
Sample 7. Screen-Capture Video of Voice Recognition Software Interpreting a Pre-recorded
Performance of Sample 3 Text. Note: To view this video, open the supplemental file to this article
from the table of contents of this issue of JCACS (Volume 14, Number 1) or visit
https://vimeo.com/188163815.
Once a text begins its iterative journey, it is never finished or complete; indeed,
through selective editing, recycling, and re-processing, the text takes on a variety of
different readings. From an educator’s perspective, this is precisely what we hope to
encourage in students when teaching critical digital literacies. In addition, we puncture the
superficial application of computers to witness the extrusion of deep layers of data available
to device-level intelligence, thereby engaging creatively with the algorithmic, deep learning
characteristics of contemporary language recognition software. We acknowledge the
artificial intelligence entities who, once trained, begin ipso facto to train the user, to
interpret them and reflect back the interpretation as search results, or word suggestions, or
formatting expectations, or in response to a command based verbal dialogue. Programs
begin to write the user, making many formal operations of textual production and delivery
first facile and then redundant. The more digital automation does for the user, the less the
user is aware of what the automation is doing, how well or how poorly it is doing it, and to
what ends, other than the most superficial aspect of serving an (inter)personal
communication.
Glitch Pedagogy
JCACS
123
Conclusion
Glitch pedagogy helps the learner to gain an overview, even if very partial and with
all manner of bias, of the big data substrate on which contemporary voice and visual
recognition systems have taken root. Through myriad network/media channels, this
substrate is brought to the surface, recycled by the users who are grown on that substrate
of global sociolinguistic behaviour, conditioning language use and awareness of the
youngest populations, as we increasingly turn toward digital devices like computers and
tablets to entertain, educate, and otherwise babysit biological offspring. As Schaff and
Mohan (2014) claim,
parents are using their digital devices as pacifiers or babysitters...as a global society
we are exposing children to digital games at a very young age. Infants in strollers
are learning content and skills using the devices provided by their parents...it would
be logical to assume these children would grow up craving the same learning
delivery method they used as an [sic] infants. (p. 16)
The point at which the computer application becomes agentive in not only the product but
also the production and assembly of communication and cognition, or semiosis and psyche,
a strong and peculiar bond is formed between the artificial other and the individuals it
serves. In this sense, the prescriptive programming of social algorithms has the same force
as genres do, as Schryer (1999) states, “because they exist prior to their users, [genres]
shape their operators; and yet their users and their discourse communities constantly
remake and reshape them” (p. 81). This reshaping process involves misappropriation of
socio-cultural/algorithmic norms. Glitch pedagogy can be used to interrogate that
relationship between served and server, between used and user, dialogically and creatively.
It opens the parameters of our own understanding, offering surprising resonant clusters of
meaning, expression made possible only through collaborative expression between human
and robotic semiosis. This way, poetry grows through the cracks in intended meaning
enlightening us as to the nature of our discourses.
In another sense, these experimental discourses with algorithmic familiars is a game.
Digital gaming is frequently put forward as a way to motivate learners “in a safe virtual
environment where failure is a powerful learning experience without the serious
consequence or stigma of real-life mistakes...in the traditional classroom, students make
mistakes and they are marked down, lose points, or they fail” (Schaff & Mohan, 2014, p.
17). Glitch pedagogy not only instigates the game-sense of learning but celebrates mistakes
and processing errors as central to creativity, inquiry, invention, and discovery of processes
underlying knowledge construction and mobilization in the twenty-first century. We provide
the following manifesto of glitch pedagogy as a provocation to curriculum thinkers and
designers and to instigate a new method of educational inquiry. With it, we hope to instigate
a different vision of the educational affordances of digital devices that play so significant a
role in contemporary learning and understanding.
Peña & James
JCACS
124
Sample 8. Manifesto of Glitch Pedagogy
References
Armstrong, S. (2015, January). Motivated value selection for artificial agents. Paper
presented at the 2015 AAAI Workshop, Austin, TX. Retrieved from
http://www.aaai.org/ocs/index.php/WS/AAAIW15/paper/viewFile/10183/10126
Aoki, T. T. (1999). Toward understanding “computer application.” Counterpoints, 70, 168–
176.
Bright, J. R. (1958). Automation and management. Boston, MA: Harvard Business School.
Chunn, E. (2014). The technological humanity of John Dewey (Bachelor of Arts honors
thesis). Portland State University, Portland, Oregon. Retrieved from
http://pdxscholar.library.pdx.edu/honorstheses/31
Deng, L., Li, J., Huang, J. T., Yao, K., Yu, D., Seide, F.,…Acero, A. (2013). Recent advances
in deep learning for speech research at Microsoft. In 2013 IEEE International
Conference on Acoustics, Speech and Signal Processing (pp. 8604–8608). Vancouver,
BC. Retrieved from http://doi.org/10.1109/ICASSP.2013.6639345
Dewey, J. (1934). Art as experience. New York, NY: Balch.
Elbow, P. (1981). Writing with power: Techniques for mastering the writing process. New
York, NY: Oxford University Press.
Emig, J. (1971). The composing processes of Twelfth Graders. Urbana, IL: National Council
of Teachers of English.
Froomkin, D. (2015). The computers are listening: How the NSA converts spoken words into
searchable text. The Intercept. Retrieved from
https://theintercept.com/2015/05/05/nsa-speech-recognition-snowden-searchable-
text/
Glitch Pedagogy
JCACS
125
Gaver, W. W. (1991). Technology affordances. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (pp. 79–84). New York, NY: ACM. Retrieved
from http://doi.org/10.1145/108844.108856
Photojojo. (n.d.). Get glitchy with your phone’s Panorama function. Retrieved from
http://content.photojojo.com/inspiration/photo-projects/how-to-make-phone-
panorama-glitches/
Gibbs, S. (2016, March 30). Microsoft’s racist chatbot returns with drug-smoking Twitter
meltdown. Retrieved from
http://www.theguardian.com/technology/2016/mar/30/microsoft-racist-sexist-
chatbot-twitter-drugs
glitch, v. (n.d.). OED online. Oxford University Press. Retrieved from
http://www.oed.com.ezproxy.library.ubc.ca/view/Entry/78999
Hernández, P. (2012). It’s not a glitch. It’s a feature. It’s art. It’s beautiful. Retrieved from
http://kotaku.com/5933722/its-not-a-glitch-its-a-feature-its-art-its-beautiful
Hickman, L. A. (1990). John Dewey’s pragmatic technology. Bloomington: Indiana
University Press.
Hickman, L. A. A., & Alexander, T. M. M. (Eds.). (2009). The essential Dewey: Ethics, logic,
psychology (Vol. 2). Bloomington: Indiana University Press.
Huang, X., Acero, A., & Hon, H.-W. (2001). Spoken language processing: A guide to theory,
algorithm, and system development. New York, NY: Prentice Hall.
International Data Corporation. (2014, February 24). Android and iOS squeeze the
competition, swelling to 96.3% of the smartphone operating system market for both
4Q14 and CY14, according to IDC. Retrieved from
http://www.idc.com/getdoc.jsp?containerId=prUS25450615
James, K. (2015). Mapping critical media literacy onto iterative remix practices. In P.
Schmidt & A. Lazar (Eds.), Reconceptualizing literacy in the new age of
multiculturalism and pluralism: A tribute to Peter Mosenthal (2nd ed.) (pp. 125–150).
Charlotte, NC: Information Age Publishing.
Johnson, B. (2011, October 31). Teaching and learning: Using iPads in the classroom.
Retrieved from http://www.edutopia.org/blog/ipad-teaching-learning-apps-ben-
johnson
Juang, B., & Rabiner, L. R. (2006). Automatic speech recognition: A brief history of the
technology development. Encyclopedia of Language & Linguistics (2nd ed.) (pp. 806–
819). Oxford, England: Elsevier.
Krapp, P. (2011). Noise channels: Glitch and error in digital culture. Minneapolis: University
of Minnesota Press. Retrieved from
http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10519768
Krutz, D. E., & Lutz, M. (2013). Bug of the day: Reinforcing the importance of testing.
Publications from the 2013 IEEE Frontiers in Education Conference (FIE) (pp. 1795–
1799). Retrieved from http://doi.org/10.1109/FIE.2013.6685147
Larson, M., & Jones, G. J. F. (2012). Spoken content retrieval: A survey of techniques and
technologies. Foundations and Trends in Information Retrieval, 5(4–5), 235–422.
Retrieved from http://doi.org/10.1561/1500000020
Leonard, D. (2013, October 24). The iPad goes to school. Retrieved from
http://www.bloomberg.com/bw/articles/2013-10-24/the-ipad-goes-to-school-the-rise-
of-educational-tablets
Lowe, J. (n.d.). When panorama photography goes wrong. HUH Magazine. Retrieved from
http://www.huhmagazine.co.uk/8050/when-panorama-photography-goes-wrong
Maddox, B. (2015). Inside the assessment machine: The life and times of a test item. In M.
Hamilton, B. Maddox, & C. Addey (Eds.), Literacy as numbers: Researching the politics
and practices of international literacy assessment (pp. 129–146). Cambridge, MA:
Cambridge University Press.
Menezes, J. (2010). Creative processes in free improvisation (Unpublished masters thesis).
Peña & James
JCACS
126
University of Sheffield, Sheffield, England. Retrieved from
http://hdl.handle.net/10174/6051
Menkman, R. (2011). Glitch studies manifesto. In Video vortex reader II: Moving images
beyond YouTube (pp. 336–347). Amsterdam, The Netherlands: Institute of Network
Cultures.
Mullis, E. (2009). The device paradigm: A consideration for a Deweyan philosophy of
technology. The Journal of Speculative Philosophy, 23(2), 110–117.
Murphy, M. E. (2014, August 5). Why some schools are selling all their iPads. The Atlantic.
Retrieved from http://www.theatlantic.com/education/archive/2014/08/whats-the-
best-device-for-interactive-learning/375567/
Neff, M. (n.d.). Did I stutter: iOS panorama glitch photos. Retrieved from
https://www.flickr.com/photos/army-of-neff/sets/72157632998618461
Nelson, A. J. (2007). Reading photobooks: Narrative montage and the construction of
modern visual literacy (Unpublished doctoral dissertation). University of Minnesota,
Minneapolis. Retrieved from http://search.proquest.com/docview/304822999
Noble, D. F. (1986). Forces of production: A social history of industrial automation. Oxford,
England: Oxford University Press.
Noble, S. U., & Roberts, S. T. (2016). Through Google-colored glass(es): Design, emotion,
class, and wearables as commodity and control. Media Studies Publications, 13.
Retrieved from http://ir.lib.uwo.ca/commpub/13
Norman, D. A. (1988). The psychology of everyday things. New York, NY: Basic Books.
Norshteyn, Y. (Director). (1975). Hedgehog in the fog [animated film]. Russia:
Soyuzmultfilm.
Oral, S. B. (2013). Exploring the ideal of teaching as consummatory experience. Education
and Culture, 29(2), 133–158. Retrieved from http://doi.org/10.1353/eac.2013.0010
Pogue, D. (2010, December). Talk to the machine: Speech-recognition programs are no
longer clumsy exercises in futility. Scientific American, 303, 40.
Pressing, J. (1988). Improvisation: Methods and models. In J. A. Sloboda (Ed.), Generative
processes in music (pp. 129–178). Oxford, England: Oxford University Press.
Rost, M. (2011). Teaching and researching listening (2nd ed.). London, England: Routledge.
Schaaf, R., & Mohan, N. (2014). Making school a game worth playing: Digital games in the
classroom. Thousand Oaks, CA: Corwin.
Schön, D. A. (1992). The theory of inquiry: Dewey’s legacy to education. Curriculum Inquiry,
22(2), 119–139. Retrieved from http://doi.org/10.2307/1180029
Schryer, C. F. (1999). Genre time/space: Chronotopic strategies in the experimental article.
JAC: A Journal of Composition Theory, 19, 81–89.
Shaw, R., & Bransford, J. (Eds.). (1977). Perceiving, acting, and knowing: Toward an
ecological psychology. Hillsdale, NJ: Lawrence Erlbaum.
Skopec, D. (2004). Digital layout for the Internet and other media. New York, NY: AVA
Publishing.
Stimson Jr., T. (1961, May). Are we licking our missle fizzle? Popular Mechanics, 115(1),
238.
TeachThought Staff. (2014, December 14). 52 of the best apps for your classroom in 2015.
Retrieved from http://www.teachthought.com/the-future-of-learning/technology/52-
of-the-best-apps-for-classrooms/
Tiles, J. E. (1990). Dewey. New York, NY: Routledge.
Willis, J. (2007). The gully in the “brain glitch” theory. Educational Leadership, 64(5), 68–73.
Yu, D., & Deng, L. (2015). Automatic speech recognition: A deep learning approach. London,
England: Springer.
Glitch Pedagogy
JCACS
127
Endnotes
1
The experiments discussed in this paper were conducted in, and used the technical resources of, the
Digital Literacy Centre of the Department of Language and Literacy Education at the University of
British Columbia.
