Interpreting?? digital?? images?? beyond?? just?? the?? visual:??
crossmodal?? practices?? in?? medieval?? musicology??
Oxford?? e-‐Research?? Centre,?? University?? of?? Oxford?? ??
Oxford?? e-‐Research?? Centre,?? University?? of?? Oxford?? ??
Eric?? T.?? Meyer??
Oxford?? Internet?? Institute,?? University?? of?? Oxford??
We argue that high-resolution naturalistic digital images of physical objects are oriented to in a very
different manner than other visual representations such as ‘inscriptions’ which are manufactured by
black-box devices in order to transform phenomena into diagrams, or ‘rendering practices’ where
scientists visually transform the meaning of objects and events using repre- sentational techniques to
select information and simplify its presentation. We show that medieval music scholars engage with high-
resolution images of physical objects through crossmodal practices relying upon the intercon- nected
senses to examine a variety of properties held within physical objects when they are displayed within
crossmodal practice, visual culture, digital humanities, medieval music
The scholarly study of medieval music manuscripts has traditionally required that musicologists
travel to libraries and museums where the artefacts of interest, the complete manuscript or
fragments of manuscripts, are physically held. However, with the introduction of digital image
archives these scholars now have the ability to conduct much of their research through both the
inspection and manipulation of digital images. When digital images are used as primary
research objects skillful technical work is conducted in order to draw out and expose visual
information that was not previously accessible. This kind of work is conceived of as
encompassing a set of scientific practices especially when manipulating, marking, categorizing
and comparing features that reside within images using graphics editing software. In addition to
these obvious practices where computerisation is used as a way to engage with information, we
also find that medieval musicologists engage in rather non- obvious practices related to the use
of technology to make sense of a physical object’s attributes as depicted within digital images
that goes beyond the visual. We find that the scholars draw upon a number of different
modalities to interpret what might be the tactile qualities of a manuscript’s parchment weight
and texture as well as the auditory qualities represented in musical notation that denotes
melody, rhythm and pitch. In this way, other sensory modalities are recovered and integrated
into their visual analysis of digital images making activities around image interpretation and
visual practice a broadly concerted effort involving not just a single sense but many senses.
The?? interconnected?? senses??
The concept of ‘multisensory experience’, and research related to it, is concerned with
developing an understanding of the role the senses play in the mediation and interpretation of
phenomena. Research in this area investigates the integration of semantics with the other senses
to explore the “different points of apprehension, interpretation [and] knowledge construction”
that make up the “dynamic cross-relationships” between the senses (Porcello et al. 2010, 57).
Interest in this area of research has existed in anthropology for some time beginning with
investigations into culturally-specified ‘sensory codes’, which are the socially-acquired
meanings attributed to sounds, smells, tastes and touch (Levi-Strauss 1969, 147-63).
Investigations into sensory codes continues to be an important research topic within
anthropology and sociology (Howes 1991; Classen 1997; Low 2005; Pink 2009) as well as for
the interdisciplinary field of sensory studies (Sensory Studies 2011) where scholars investigate
sensory experiences in a wide variety of settings in order to understand how their socially-
acquired meanings inform perceptual understanding.
More recently, within Science and Technology Studies (STS) scholars have discussed the
manner in which scientists engage in very specific types of ‘body-work’ when participating in
discourse about experimental data and making models (Myers 2008). Additionally, the non-
visual dimensions of embodied knowledge have been discussed specifically in relation to
the interpretation of auditory phenomena in laboratory settings (Mody 2005). Also, within
Computer Supported Cooperative Work (CSCW), embodied actions such as gesture, gaze and
bodily orientation are investigated to understand how communication, collaboration and
interaction might be mediated through computational devices (Suchman 1993; Robertson 1997;
Luff et al. 2000). Finally, ethnomethodologically-informed studies of work examine the
embodied and collaborative practices that facilitate the mutual understanding of sensory
experiences such as texture and taste when producing materials in a laboratory setting (Goodwin
In the research area of the ‘digital humanities’ (Schreibman et al. 2004), technologies are
being designed that integrate computational tools and techniques with the more traditional
scholarly activities found in history, philosophy, literature, art, dance and music. As a result,
computing activities are embedded alongside the analytic, critical and speculative methods
employed by scholars in the analysis of texts, artefacts, movements and sounds. And so, as
scholars’ research objects increasingly become digitised and made available within web-based
digital archives and virtual workspaces, we find that some of their research practices are being
transformed as well (ibid; Terras 2006, de la Flor et al. 2010b). As part of a programme of
empirical research studying scholarship in transition as new technologies are taken up across a
variety disciplines (Jirotka et al. 2005; de la Flor et al. 2010a, de la Flor et al. 2010b), this paper
discusses how new computational technologies are transforming musicologists’ research
practices through the manipulation and inspection of digital images which serve as proxies for
the inspection of actual physical manuscripts. So rather than relying upon simulations or
visualisations of phenomena and objects, these scholars rely upon the study of high-resolution
digital images that depict the actual physical objects themselves. While musicologists’
increasingly use digital images to study physical objects, we investigate the ways in which this
affects both the reading and interpretation of medieval music manuscripts.
We find that musicologists engage in the obvious computing activities when they use image
editing software to manipulate, mark, categorize and compare features that reside within digital
images (activities associated with scientific practice). However, we also find that they engage in
rather non-obvious practices as they draw upon the interconnected senses, the visual, auditory
and tactile, to examine the physical properties of objects (their condition) and the features found
within them (their content). In this way, musicologists’ analysis of objects and features
displayed within digital images is mediated through, not only the technology itself, but also
through their attentiveness to their own inter-related senses in the interpretation of the properties
and qualities of an object. So within the wider, more general discussion of visual culture, we
can say that these practices are unlike the use of ‘inscriptions’ that are manufactured by black-
box devices in order to transform phenomena into diagrams and where the markings are then
translated by scientists to support of scientific discourse (Latour and Woolgar 1979), and also
unlike ‘rendering practices’ where scientists visually transform the meaning of objects and
events using representational techniques to select information and simplify its presentation in
order to construct a more visible, knowable or ‘docile object’ (Lynch 1985, 43). In this paper
we discuss how medieval music scholars engage in crossmodal practices where other sensory
modalities are recovered and integrated into their visual analysis of digital images making
activities around image interpretation and visual practice a concerted effort involving not just a
single sense but many senses.
On the one hand, digital images afford a kind of durability that enables detailed visual
inspection and manipulation which would otherwise be impossible (e.g. adjusting a
manuscript’s gamma levels and extreme magnification) or that would permanently damage or
destroy the physical object (e.g. digitally peeling away features to reveal objects underneath).
On the other hand, musicologists orient to visual images beyond their assumed exclusivity as
“observational and objectifying tools” (Pink 2009, 99), where the visual is isolated from the
other senses, and instead draw upon the integrated senses (visual, tactile and auditory) to
interpret the condition and content of medieval music manuscripts. Assertions of
‘ocularcentrism’ (the privileging of vision over the other senses) in the investigation and
reporting of both the material practices of visual cultures and in the analysis of a community’s
practices, knowledge production and socially-acquired meanings has been raised as an issue
across interdisciplinary boundaries (Howes 1991; Classsen 1997; Pink 2009; Porcello et al.
2010). Because the semantic and the visual are the most obvious modes of conveying
understanding and information these tend to be the focus but they may not necessarily deliver
the entire story of how people make sense of their world (Low, 2005). As our experiences are
increasingly mediated through digital technologies and where currently our primary route of
access into virtual reality/research spaces is through the visual, the nature of embodied
experiences as one participates within those spaces needs to be taken seriously. These hybrid
interaction spaces have been found by some researchers to be places where “the visual is treated
as inseparable from the tactile, kinesthetic, and proprioceptive” (Porcello et al. 2010, 58-9). We
have also found that the visual is deeply interconnected with the other senses as medieval
musicologists inspect digital images using computational resources alongside the analytic,
interpretative and speculative techniques they rely upon to understand the physical objects that
Methodology?? and?? approach??
To investigate embodied practices around digital image analysis we conducted interviews using
a semi-structured format with seven medieval music scholars distributed across three continents
(Europe, North America and Australia). From senior scholars who began their careers in the
early 1960s to emerging scholars just starting out in the 21st century, informants consider
themselves to be enthusiastic early adopters of digital technologies in relation to their research
In addition to interviews, we facilitated one focus group with five scholars to discuss the
ways in which digital images are used in the analysis of medieval music documents. Finally, we
conducted two ethnographic fieldwork sessions with a pair of medieval musicologists, a scholar
and her student, to acquire a more detailed understanding of how they use digital images as part
of their routine research practice. During the fieldwork phase, audio-visual elicitation
techniques were utilised to discuss selected video fragments with the scholar and her student in
order to understand both the details of activities as well as the technical language they used. We
draw upon ethnomethodology (Garfinkel 1967) and an analysis of conversation (Sacks et al.
1974) in the study of the video recordings taken during observational fieldwork. This
exploratory research study took place over a 12-month period from early 2010 to early 2011.
For this paper we refer to the interview data to discuss how computing activities enable the
detailed inspection and manipulation of digital images. After which we draw upon video
fragments taken from fieldwork studies that reveal how interpretative activities related to
‘computational seeing’ are interconnected with other senses specifically where the
musicologists attempt to recover tactile and auditory understandings related to the physical
manuscripts displayed within digital images.
Medieval musicology is the study of music composed and performed between the 6th and 15th
century, from approximately 500-1400 (cf. Hoppin 1978). Almost all of the extant evidence of
music from this period comes from handwritten manuscripts produced by scribes. Very few of
these manuscripts still exist in their complete form, although more survive as damaged and
fragmentary pieces. In many cases, fragments have been preserved only because they were re-
used as bookbinding material or for other purposes such as chair stuffing, papering for ceilings,
hat-box linings and padding for musical instrument cases. Medieval musicologists are interested
in these source manuscripts for a variety of reasons. For instance, the content of these materials
enable scholars to study both secular and sacred music within a specific time period and draw
out its relationship to historical events. They also compare compositional styles and notational
variation, as well as conduct analyses of song in courtly contexts to understand how it may have
been used to communicate political ideas and create coherence within a community.
Additionally, scholars attempt to reconstruct fragmentary pieces with reference to other
manuscripts and fragments. Finally, the layout and notational style of manuscripts is of great
interest for understanding how to read and actually perform these musical works today.
Traditionally, medieval musicologists conduct their work by analysing the original physical
manuscripts (Fig. 1), using a magnifying glass to inspect details and a ruler to measure its size.
They also inspect physical characteristics such as leaf textures, bookbinding, and ink pigments.
Additionally, musicologists are interested in understanding the sounds that a series of musical
notes might represent. They may also reference other materials associated with a manuscript
such as facsimile books (high-quality photographs) or microfiche (black and white or colour
negatives approx. 25 times smaller than the original manuscript’s size accessed through a
reading machine and then magnified).
Medieval musicologists inspect both the content and condition of these manuscripts and
increasingly high-resolution digital images of them are being made available through digital
image archives. In many cases, archives such as The Digital Image Archive of Medieval Music
(DIAMM 2011) and others, were originally set-up to support the conservation of medieval
polyphonic music manuscripts to ensure that their contents may still be viewable in cases of
their loss from natural degradation, theft and geopolitical difficulties to gaining access.
However, increasingly musicologists are using these online archives as research tools. In what
follows, we discuss how medieval musicologists orient to the use of digital images when
conducting research related to the analysis of medieval manuscripts and fragments.
Figure 1. Digitized medieval manuscripts, the DIAMM Handbook.
Inspecting?? and?? manipulating?? digital?? images??
Within the DIAMM archive, for instance, each image includes the following elements: a
manuscript page or fragment, a colour patch and ruler (Fig. 1). Both the colour patch and the
ruler are placed alongside the manuscript item serving as reference points that provide
indications of its actual physical appearance: these are used by scholars to orient to physical
objects as they are portrayed in digital form. The ruler provides a sense of the item’s actual size
and the industry-standard colour patch assists in determining if a scholar’s computer monitor is
calibrated correctly ensuring that they see colours accurately.
Through this display, each leaf or fragment of a manuscript is extracted from its context
within a material codex and its surrounding pieces and is presented as a discreet scientific
specimen. This is particularly interesting in light of two well-known accounts within STS of
scientists who also use visual representations of physical objects. One has characterized their
use as leading to, in the case of astronomy, a transformation of “observational field science to
an image-processing laboratory science” (Knorr Cetina, 1992, 117), and in the case of the UK
systematics community, as provoking resistance where existing material culture influenced
negative responses to an engineered virtual culture (Hine 2006).
Both of these concerns - the transformation of fieldwork approaches into a laboratory-type
science, and points of resistance to the introduction of new technologies are important topics in
debates around the introduction of computational imaging within a range of research
disciplines. In our study the researchers we interviewed appraise the properties of digital images
as facilitating and enhancing their research. For instance, one informant noted that he can:
Probably do 80, 90% of everything through the high-res digital images and only for the remaining 10,
15, 20% you need to see the manuscript. (Medieval musicologist A)
Prior to the availability of digital image archives, gaining access to source materials was once,
and often still is, difficult especially in cases where the manuscript is severely degraded and
handling might destroy it further, or when strict requirements are imposed on independent
scholars who are not affiliated with an institution. Also, when scholars are based in continents
far from Europe (where many the physical objects are held) some rely upon digital images as
their primary research objects. In one case, a scholar we interviewed has tailored his research
towards investigations of the scribal process focusing specifically on the study of erasures
which can be detected through digital image manipulation. This medieval musicologist tells us
The traditional ways of detecting erasures is actually getting in there and consulting the manuscript.
Now that is increasingly becoming more difficult … it was very useful to actually have the color
images to work with because something the computer technology allows someone to detect are
erasures through the manipulation of the image gamma. That’s the image gamma or the images levels,
one can actually bring out what would normally be invisible, literally invisible on the page, to
essentially a very clear indication that there’s an erasure there. (Medieval musicologist G)
In one manuscript, the scholar discovered that the scribe made a substantial erasure at a point
that dramatically changed the meaning of the musical notation. This allowed him to form an
entirely new reading of the scribe’s activity which was instrumental in forming a new
interpretation of the participation of that individual in the production, history and reception of
the manuscript. The details of how to conduct technical computing activities are shared amongst
musicologists informally through ad hoc emails or in-person tutorials and also through the
distribution of the DIAMM Digital Restoration Workbook (Craig-McFeely and Lock 2006).
This workbook is widely known and distributed within the community and provides step-by-
step guidance in the implementation of different technical combinations to produce digitally
restored images through the manipulation of colour, brightness, contrast, sharpness, as well as
techniques for the realignment of objects such as staves using off-the-shelf commercial
For digital archives like DIAMM that not only store digital images but also engage in
resource creation through digital image capture activities, taking extremely high-resolution
photographs is essential to making possible the application of post-processing techniques that
make visible content that could not be seen otherwise. In addition to image manipulation,
musicologists conduct detailed comparisons of digital images to trace the careers of scribes as
they moved from place to place. As one scholar explained:
The scribes of the Middle Ages worked really hard to be anonymous. If somebody started taking over
the work of another scribe on page 50, the scribe would copy the handwriting of the other person.
They would try not to use their own. And so you end up looking for very, very small, subtle, telltale
sort of habits that you can only see with really great quality photographs, and then I’m able to see how
various manuscripts connect with each other. (Medieval musicologist E)
Additionally, because the physical manuscripts and fragments written by the same scribe are
often geographically distributed, those that have been digitized and made available within
digital archives make it possible for scholars to compare them across various libraries and
museums without the need for extensive travel which may otherwise take many months to
complete at considerable expense. In this way, historical overviews of a particular period are
accelerated as another musicologist confirms with an example from the 16th century (there is
interest in manuscripts written immediately after the medieval period because their content
includes medieval music):
Say I want to look at music manuscripts in the first decade of the 16th century. I can now pull it up
because DIAMM metadata has a date field, so I said between 1500 and 1510, and I get a list of 80
manuscripts across Europe which were produced between 1500 and 1510. And then with some ease I
can either look at the images in DIAMM or I can find the images elsewhere. And so give me a week
or so and I can give you that overview and then you can search that in all sorts of ways. Or on your
screen you can actually put an image from a manuscript from Cyprus and an image from a manuscript
from St. Petersburg, an image from a manuscript from St. Andrews side by side. And that way you
can see things, you can draw direct comparisons which was, I tell you, it was not always possible and
what people did with photography, with microfilm, with just a lot of traveling. But it’s now come
within reach of a lot more people. (Medieval musicologist A)
This accessibility has in some way democratized scholarship in medieval musicology where
those in universities who may not have funding to undertake long excursions, independent
scholars not affiliated with an institution and choir enthusiasts have in some, but not all cases,
free access to high-resolution photographs of source materials. However, even with the
acceleration of research facilitated through ease of access to digital image archives, one scholar
talked about the continued pressure to work in more traditional ways:
I do feel pressure, for instance there’s this notion in the field that you can always get more out of
seeing the original than seeing a digital image of it. And I do feel pressure to work more with
originals than with the digital images because of the traditions in the field. Whereas frankly, there are
times that that’s really important, but for the most part I do feel like I get more out of sitting, using
these images on my computer. I do feel like there’s a certain pressure that that’s not what top scholars
do because that’s not what top scholars did 25 years ago. (Medieval musicologist E)
The tension between traditional and emerging forms of research practice is tied to the
expectation that serious scholars actually handle and view physical manuscripts as a senior
You obviously need to have seen the physical thing to get a sense of it. And yeah, there are things that
you can only see on the original, including for example, the structure of the manuscript, the way the
leaves are folded and sewing and that kind of thing. (Medieval musicologist B)
Whilst recognising this tension, scholars nevertheless acknowledge the new analytic
possibilities digital images provide:
Digital imaging opens up the possibility for manipulation in ways that can restore stuff that is no
longer visible. For me the magnification, extreme magnification is very useful. Being able to zoom in
much further than you could even with a magnifying glass. Often, it allows me to see things I couldn't,
that I wouldn't be able to read or that I can read better from a massively magnified image. (Medieval
In addition to the benefits, these scholars are also keenly aware of the limitations of digital
images. Firstly, they cannot appreciate the whole book as an object in its own right when
viewing a series of separate images of each of its pages. These discrete images have been
extracted from their context within a codex making it difficult for musicologists to get a sense
of a book’s construction especially how the pages are gathered into quires. Knowing the details
of its construction enables scholars to tell if a page was part of the original manuscript or if a
new page was inserted into the original leaves. Even though recent manuscript catalogues
usually list this information, the majority of manuscripts were catalogued many decades ago,
often in the 18th or 19th century, making it such that these details are obtainable only through
handling the manuscript itself. Secondly, scholars rely upon their sense of touch to feel the
parchment (or paper in later manuscripts). The thicker and rougher it is, the cheaper it would
have been. Conversely the thinner and smoother it is, the more expensive. Knowing these
details may reveal the status of the music itself when it is associated to the quality of the
parchment that it was written upon. In addition it might also suggest the likely context of its
Also, when viewing digital images scholars can’t really distinguish between a mark that is
the result of scraping or rubbing (e.g. an attempt to erase something previously there) or a mark
that indicates damp or some other stain. They usually determine this by feeling the parchment’s
texture where such marks might tell them something interesting about the manuscript’s history.
Finally, when examining manuscript fragments scholars may be able to spread the different
segments out upon a table to try to fit them together and determine what pieces might be
missing. This is more difficult to do with digital images because aspects of the feel of the
parchment as well as the shape and size of the pieces would come into play. In regards to
making sense of information associated with the sense of touch one scholar tells us that:
We can appreciate this partly through images, you saw how we can sometimes tell the parchment is
thin from seeing the show-through of ink from the other side, but it’s much easier to appreciate this
difference if you can touch it. (Medieval musicologist F)
This reliance upon senses other than, and in addition to, the visual in the interpretation of
physical objects has implications for scholars who investigate those objects through digital
images. In the following section, we discuss some of these implications drawing upon audio-
visual fragments taken from our observations of medieval musicologists at work. We reveal
how the scholars employ what we refer to as ‘crossmodal practices’ in their analysis of physical
objects as they use digital images to interpret those objects. First, we show how they formulate
workarounds to discern what might be tactile qualities of a manuscript’s parchment and second,
how they make sense of notational details written into manuscripts through talk about its
rhythmic behaviour; the way it might sound and the way it may have been meant to be heard.
Discerning?? the?? physical?? qualities?? of?? a?? manuscript??
Some have described sensory recollections (e.g. remembering the specific tactile feelings and
qualities of an object) as an avenue into the ‘sensory imagination’ used to ignite one’s memory
in order to re-experience specific sensations that a photograph, object, setting or event may
conjure up. These are often treated as abstract cognitive processes that only the imaginer can
know, also referred to as the ‘embodied imagination’ it may only be accessed through self-
reporting mechanisms such as interviews or first-hand phenomenological accounts (Ihde 1976;
Pink 2009). However, we find that what are often framed as cognitive phenomena, may also be
made visible through observation of the practices and interactions that occur within social
settings (cf. Lynch 2006). In the example below, a medieval musicologist (Pam) and her student
(Jeremy) participate in the inspection of a digital image so that they might recognise and
evaluate visual depictions that could provide evidence related to the tactile qualities of a
Pam: There’s terrible show-through of the decoration from the other side
there ((pointing to the area))
Um, which suggests that the parchment is very thin.
((rubbing her fingers together))
But we’d actually have to touch it to verify that.
Figure 2. Teaching the skills necessary to interpret visual traces within digital images.
Pam points to the visual features within the digital image to highlight the markings that suggest
the parchment’s thinness. By rubbing her fingers together, she makes visible to her student the
activity associated with feeling parchment texture. Interpreting what the show-through of inks
might indicate within the digital image enables the pair to recognise and to speculate about
sensory qualities other than the visual in their analysis of the manuscript. From this activity the
student learns how to interpret ‘show-through’ enabling him to become sensitive to what might
be the tactile qualities of the physical object.
During the activity, the scholar remains mindful of the tension between the inspection of
digital versus physical objects and so adjusts her initial analysis with a cautionary note that the
parchment would need to be physically touched to verify this interpretation of visual traces. We
see how the visual can be interconnected with the other senses in the manner that Pam instructs
Jeremy specifically in the application of the ‘sensory imagination’ as they orient to visual traces
in a way that allows them to speculate about the parchment’s texture. This strategy, interpreting
‘show-through’, is a workaround formulated by the scholar when only a digital image of the
physical object is available. Such constraints invite us to consider other ways they might
recognise and evaluate visual depictions in their analysis of digital images.
Distinguishing?? smudges?? from?? holes?? within?? the?? parchment??
Scholars told us during interviews (see above) that distinguishing marks which appear on a
manuscript’s parchment as either a scribe’s erasure which may result in a hole developing over
time, or some kind of stain, requires that they feel its texture. However, when viewing digital
images of such marks the scholars we observed apply a workaround to distinguish a smudge
from a hole when they cannot actually touch the physical object. Because digital images present
three-dimensional objects in two-dimensional space they flatten the appearance of physical
objects and so the scholar and her student develop a novel approach for interpreting marks
within these flat images:
Jeremy: I'm going to take it that these markings aren’t- are just marks rather
than anything else. Cause they don't appear in this chant book, so-
Pam: Should we just zoom right in and see if we can see those-
>wait for the resolution to sort itself out<.
They might even be holes might’en they (.) some of them.
Jeremy: Yea, if we look at the other side we might be able to see if=
Pam: =There's a hole in the equivalent place.
Jeremy: Yup, which would be on the left-hand side.
Pam: Yea, good thinking.
Figure 3. Developing a workaround to distinguish marks that appear on the parchment.
Jeremy identifies markings on the parchment which appear to be something other than musical
notes. Pam suggests they zoom into the area to get a closer look proposing that the marks may
even be holes. The pair then collaboratively formulate a way of making the mark more
intelligible with Jeremy suggesting (“if we look at the other side we might be able to see if-”),
and Pam interjecting (“there’s a hole in the equivalent place”). During this exchange they
develop a workaround that may help them to distinguish a smudge from a hole. Upon visually
inspecting the respective digital images taken of each side of the same parchment leaf they hope
to see whether or not a feature on one side is visible at the same place on the other side of the
Jeremy: I can’t see any visible-
Pam: (It) would be somewhere wouldn’t it
Pam: Mm, ok so it’s a smudge rather than a hole (.) Some kind of stain.
There’s obviously considerable staining=
Jeremy: =All over=
=In various parts. And also quite a lot of rubbing of some of the inks.
As they inspect the digital image of the other side of the parchment leaf, Jeremy and Pam agree
that there is no visible mark at the same position. Pam concludes that it is a smudge rather than
a hole and bolsters her assessment by saying (“there’s obviously considerable staining”). These
other unambiguous markings indicate staining in other areas and so provides Pam and Jeremy
with supporting evidence. As in the previous example, the musicologists collaboratively
strategize to develop a workaround for inspecting three-dimensional objects as they appear
within the flattened two-dimensional space of digital images. Identifying a mark as a smudge or
a hole within this virtual workspace requires that they compare two digital images of the same
leaf (front and back) and in this case, the absence of a matching mark on the other side indicates
something else about its texture, that the parchment has not been perforated and so it is not an
erasure made by a scribe but rather a smudge or stain which has entirely different implications
for their interpretation of the manuscript.
Making?? sense?? of?? musical?? symbols??
The scholar and her student make assessments about physical objects (parchment leaves) with
digital images by strategising workarounds. They also use them to gain access to the music
written upon the leaves where digital images provide access to temporal phenomena written as
auditory symbols (musical notes) allowing the musicologists to translate a sequence of notes
into sounds that convey their rhythm and melody. The complexity of translating musical
symbols into sounds has been described as a cognitive process where musicologist and
musicians use their ‘inner hearing’ to imagine what notes might sound like through accessing
the ‘mind’s ear’ (Bailes 2009, 41). However, when we take account of musical performance as
social practice (Cook 2001) an opportunity presents itself for us to move away from framing the
activity of translating musical symbols into sounds as a cognitive topic. This relationship
between musical notation and performance and the activity of translating symbols into sounds is
a complex activity:
Of course the student of linguistics uses the art of speech to study speech, whereas the student of
music uses speech to study another art, whose resemblances to and differences from speech are not
easy to trace. (Seeger 1951 quoting Bloomfield: 245)
Music can be talked about through verbal discourse as well as through talk about sound
qualities where vocalizations of timbre, melody and rhythm act as descriptions of musical
behaviour (Feld and Fox 1994). There is commonly an “intertwining of speaking and musicking
[at the] site of social interaction amongst musicians” (Feld et al. 2004, 322) and in our final
example, the musicologists talk about musical behaviour as they compare the rhythms of two
13th Century polyphonic pieces of the same 6th Century plainchant - Alleluia Dies Sanctificatus
(Alleluia for Christmas day).
Here, Pam and Jeremy want to understand the rhythmic differences between the two pieces
as written in the Florence and the Cambridge manuscripts1. The scholars would like to know if
they possess different rhythmic qualities and if they do what its significance might be:
would that change the way you read it do you think?
Well yea >˚cause it’s˚< da dum - da dum - da dum
yea cause it would put it in the second mode.
It's the same notes isn't it but differently ligated (.3) umm
Jeremy simultaneously taps out the pattern with his pencil while vocalizing the rhythm that he
sees. Upon completing his sounding/tapping Jeremy concludes that the notational ligature (a
device used to represent rhythm) would put that line into the second modal rhythm, meaning
that the beats would be sub-divided differently. Rhythm in medieval music is represented in
‘modal notation’ where long and short durations are determined by a note’s spatial position
within a group of notes rather than with free-standing representations used today in the form of
half or whole, quarter, sixteenth note, etc (e.g. ). In these two manuscripts a repeated or
double C is ligatured (spaced apart) differently.
Upon first glance the same notes within the two manuscripts would be read/sounded/heard
differently. Pam then turns to the Cambridge manuscript to determine what its rhythmic pattern
Pam: Yea >or as he put it here< dum da dum - da - dum da dum -
da - dum da dum.
Figure 4. Sounding the rhythmic pattern in the Cambridge manuscript.
The scholars collaboratively work through the musical behaviours of each manuscript using
both verbal discourse and soundings that are mutually understood for the purposes of trying to
make sense of the rhythmic relationships between notes. Earlier in the session Jeremy had
attempted to translate the Cambridge manuscript into modern notation and Pam asks him how
he has transcribed the line:
Jeremy: dum da dum - da - dum da dum (.) well, I didn't know what-
Pam: Some kind of rest ((laughs)) dum da dum dum
Jeremy: dum da dum dum
How does it go over there?
Figure 5. Comparing rhythmic sound patterns in the Cambridge manuscript transcription.
The pair compares Jeremy’s transcription, again through performed sounds, so that they might
recognise rhythmic patterns. Later in the session, the seeming rhythmic difference between the
two manuscripts is resolved. The scholars conclude that each scribe made different decisions for
how to represent the same rhythmic pattern. This difference can be attributed to a rule in modal
notation which states that two notes of the same pitch cannot be ligatured. So, in this case there
are two options; use no ligature at all, which is what Cambridge scribe did or switch around the
ligatures of the whole passage so one C note is attached to the note before it in a ligature and the
other C is attached to the note after in another ligature as the Florence scribe has done. The risk
of doing this is that the ligature pattern is “messed up” so that if you look at the Florence
manuscript quickly one might think it was in a different rhythm. However, when viewing that
line within the context of the entire musical piece the scholars identify it as being in the same
modal rhythm as the Cambridge copy.
The activity of rhythmic comparison would be virtually impossible to accomplish
exclusively through language about music and the use of musical terminology alone. Rather,
vocalized descriptions of musical behaviour are produced to describe sounds directly and are
intertwined with talk. In this manner, digital images of medieval music manuscripts are oriented
to so that the scholars may evaluate and make sense of the rhythmic dimensions of musical
notes as they are displayed within digital images. As musical notes lend themselves to being
read or sung or tapped aloud, the auditory dimensions of medieval music manuscript analysis is
not integrated or embedded into the digital object which may lead to considerations of how
digital images might be complimented with auditory tools in this context.
This paper has addressed the topic of ‘computational seeing’ from the perspective of the
‘sensory turn’ where we show that the privileging of vision over the other senses may need to
be reassessed in both computational interaction design as well as studies that examine
sociotechnical configurations more generally. The scholars we studied are enthusiastic about
taking advantage of digital objects. However, we find that they must strategise workarounds
when using them as research objects in their own right. The analysis of digital images that
depict physical objects is quite different to the analysis of visualisations which transform
phenomena and objects in a way that amplifies, re-represents or creates information anew either
through manual processes or by using complex mathematical algorithms.
Simulations are artificial where truth, falsehood and the imaginary are difficult to distinguish
(Baudrillard 1994) whereas high-resolution digital images of physical objects are intended to
depict ‘the real’. From this we suggest that within the continuum of the real, the neoreal and the
hyperreal, different visual cultures may emerge along with different material practices. In our
study, depicted physical objects are the primary research interest to scholars and here the notion
of visual culture and its associated material practices is expanded to include a more broadly
conceived notion of visual interpretation that involves not just a single sense but many senses
which impact upon both the reading and interpretation of medieval music manuscripts.
The ways in which medieval musicologists make sense of the properties and attributes of the
physical objects displayed within digital images raises many questions. What is gained and what
is lost when physical objects are displayed in digital form? When and how is the visual
interconnected with the other senses when inspecting familiar types of physical objects through
digital images? How are high-resolution digital images of objects interpreted if researchers have
no first-hand sensory experiences of them (e.g. Mars Rover photographs, microscopic photos or
satellite imagery)? These questions may hold relevance in research disciplines where
computational seeing involves the investigation of real-life objects in as true-to-form as possible
and where maintaining the integrity of the object within its digital depiction is critical to its
analysis in digital form.
In this case study, digital images were used to reconstitute the material and sonic properties
of physical objects. We suggest these findings contribute to a reconceptualisation of the notion
of visual culture and its associated material practices, where other sensory modalities are
recovered and integrated into the visual analysis of digital images making activities around
visual image interpretation a broadly concerted effort involving not just a single sense but many
1?? The?? manuscripts’?? names?? indicate?? the?? locations?? where?? they?? are?? currently?? archived?? and?? not?? their?? origin.?? Both??
are?? thought?? to?? have?? been?? written?? in?? Paris?? or?? northern?? France?? and?? each?? is?? closely?? associated?? with?? Notre?? Dame??
Cathedral?? where?? the?? music?? is?? said?? to?? have?? originated.??
We would like to thank the medieval musicology scholars who took part in our interviews and
fieldwork. Additionally, we thank the Research Information Network (grant RIN P28), the
EPSRC (grant EP/P503876/1), and the ESRC (OeSS grant RES-149-25-1082) for making this
Bailes, Freya. 2009. “Translating the musical image: case studies of expert musicians.” in Sounds in
translation: intersections of music, technology and society, Editied by Chan, Amy and Alistair
Noble, 41-58. Canberra: Australian National University.
Baudrillard, Jean.1994. Simulacra and Simulation, Ann Arbor: The University of Michigan Press.
Classen, Constance. 1997. “Foundations for an anthropology of the senses.” International Social Science
Journal, 153: 401-412.
Cook, Nicholas. 2001. “Between process and product: music and/as performance.” Music Theory Online,
7(2): Accessed December 19, 2011.
Craig-McFeely, Julia, and Alan Lock. 2006. DIAMM Digital Restoration Workbook. Oxford: Oxford
Select Specialist Catalogue Publications.
de la Flor, G., Ojaghi, M., Martínez, I., Jirotka, M., Williams, M., Blakeborough, T. 2010a.
“Reconfiguring practice: the interdependence of experimental procedure and computing
infrastructure in distributed earthquake engineering.” Philosophical Transactions of the Royal
Society, A, 368: 4073-4088.
de la Flor‚ Grace, Marina Jirotka‚ Paul Luff‚ John Pybus and Ruth Kirkham. 2010b. “Transforming
scholarly practice: embedding technological interventions to support the collaborative analysis of
ancient texts.” Journal of Computer Supported Cooperative Work‚ 19(3/4): 309−334
DIAMM. 2011. “About DIAMM.” Accessed December 19, 2011.
Feld, Steven, Aaron A. Fox, Thomas Porcello and David Samuels. 2004. “Vocal Anthropology: From the
Music of Language to the Language of Song.” In A Companion to Linguistic Anthropology, Edited
by Alessandro Duranti, 321-345. Oxford: Blackwell Publishing.
Feld, Steven and Aaron A. Fox. 1994. “Music and Language.” Annual Review of Anthropology, 23: 25–
Garfinkel, Harold. 1967. Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice-Hall.
Goodwin, Charles. 1997. “The Blackness of Black: Color Categories as Situated Practice.” In Discourse,
Tools and Reasoning: Essays on Situated Cognition, Edited by Lauren B. Resnick, Roger Säljö,
Clotilde Pontecorvo, & Barbara Burge, 111-140. Berlin, Heidelberg, New York: Springer.
Goodwin, Charles. 1994. “Professional Vision.” American Anthropologist, 96(3): 606-633.
Hine, Christine. 2006. “Computerization Movements and Scientific Disciplines: The Reflexive Potential
of New Technologies.” In New Infrastructures for Knowledge Production: Understanding e-
Science, edited by Christine Hine, 26-47. London: Information Science Publishing.
Hoppin, Richard, H. 1978. Medieval music. London: Norton.
Howes, David. ed. 1991. The varieties of sensory experience: A sourcebook in the Anthropology of the
senses. Toronto: University of Toronto Press.
Ihde, Don. 1976. Listening and voice: phenomenologies of sound. Albany, N.Y: SUNY Press.
Jirotka, Marina, Rob Procter, Mark Hartswood, Roger Slack, Andrew Simpson, Catelijne Coopmans,
Chris Hinds and Alex Voss. 2005. “Requirements for Collaboration and Trust in Healthcare
Innovation: The eDiaMoND Case Study.” Journal of Computer Supported Cooperative Work,
Jones, Caroline, A. ed. 2006. Embodied Experience, Technology, and Contemporary Art. Cambridge,
MA: MIT Press.
Knorr Cetina, Karin. 1992. “The Couch, the Cathedral and the Lab: On the Relationship between
Experiment and Laboratory Science.” In Science as Practice and Culture, edited by Andrew
Pickering, 113-138. Chicago: University of Chicago Press.
Latour, Bruno and Steve Woolgar. 1979. Laboratory life: the construction of scientific facts. Beverly
Lévi-Strauss, Claude. 1970. “Fugue of the Five Senses.” In The raw and the cooked: introduction to a
science of mythology, by Claude Lévi-Strauss, 147-63. London: Jonathan Cape, 1970.
Low, Kelvin, E.Y. 2005. “Ruminations on Smell as a Sociocultural Phenomenon.” Current Sociology, 53:
Luff, Paul, Jon Hindmarsh and Christian Heath. eds. 2000. Workplace Studies Recovering Work Practice
and Informing System Design. Cambridge UK: Cambridge University Press.
Lynch, Michael. 2006. “Cognitive activities without cognition? ethnomethodological investigations of
selected ‘cognitive’ topics.” Discourse Studies, 8(1): 95-104.
Lynch, Michael. 1985. “Discipline and the material form of images: An analysis of scientific visibility.”
Social Studies of Science, 15: 37-66.
Mody, Cyrus C. M. 2005. “The sounds of science: Listening to laboratory practice.” Science Technology Download full-text
& Human Values, 30(2): 175-98.
Myers, Natasha. 2008. “Molecular Embodiments and the Body-work of Modeling in Protein
Crystallography.” Social Studies of Science, 38(2): 163-99.
Parrish, Carl. 1957. The Notation of Medieval Music. New York: W.W. Norton.
Pink, Sarah. 2009. Doing Sensory Ethnography. London: Sage.
Porcello, Thomas, Louise Meintjes, Ana Maria Ochoa, and David W. Samuels. 2010. “The
Reorganization of the Sensory World.” Annual Review of Anthropology, 39: 51–66.
Robertson, Toni. 1997. “Cooperative work and lived cognition: A taxonomy of embodied actions”, Paper
presented at the 5th European Conference on Computer-Supported Cooperative Work, Lancaster,
UK, September 7-11.
Sacks, Harvey, Emanuel A. Schegloff, and Gail Jefferson. (1974). “A simplest systematics of the
organization of turn-taking in conversation.” Language, 50(4): 696-735.
Schreibman, Susan, Ray Siemens and John Unsworth. eds. 2004. A companion to digital humanities.
Oxford: Blackwell Publishing.
Seeger, Charles. 1951. “Systematic Musicology: Viewpoints, Orientations, and Methods.” Journal of the
American Musicological Society, 4(3): 240.248.
Sensory Studies. 2011. “Research Directory.” Accessed December 19, 2011.
Suchman, Lucy. 1993. “Technologies of Accountability: Of Lizards and Aeroplanes.” In Technology in
Working Order. Edited by Graham Button, 113-126. London: Routledge.
Terras, Melissa. 2006. Image to Interpretation: An Intelligent System to Aid Historians in Reading the
Vindolanda Texts. Oxford: Oxford University Press.