Developing Digital Records: Early Experiences of
Record and Replay *
Andy Crabtree, Andrew French, Chris Greenhalgh, Steve Benford, Keith Cheverst, Dan
Fitton, Mark Rouncefield and Connor Graham
Abstract. In this paper we consider the development of ‘digital records’ to support ethnographic study of
interaction and collaboration in ubiquitous computing environments and articulate the core concept of ‘record
and replay’ through two case studies. One focuses on the utility of digital records, or records of interaction
generated by a computer system, to ethnographic inquiry and highlights the mutually supportive nature of digital
records and ethnographic methods. The other focuses on the work it takes to make digital records support
ethnography, particularly the work of description and representation that is required to reconcile the fragmented
character of interaction in ubiquitous computing environments. The work involved in ‘making digital records
work’ highlights requirements for the design of tools to support the endeavour and informs the development of a
Replay Tool. This tool enables ethnographers to visualize the data content of digital records; to extract
sequences of relevance to analysis and remove non-relevant features; to marry recorded content with external
resources, such as video; to add content from internal and external resources through annotation; and to reorder
digital records to reflect the interactional order of events rather than the recorded order of events.
Key words. CSCW, ethnography, e-Social Science, digital records, record and replay.
* © Springer, 2006. This is the author’s version of the work. It is posted here for your personal use. Not for
redistribution. The definitive version was published in Computer Supported Cooperative Work: The Journal of
Collaborative Computing, vol. 15 (4), pp. 281-319.
e-Social Science is a UK initiative sponsored by the Economic and Social Research Council
(ESRC).1 The initiative aims to foster the development of new technologies to support a
broad range of social science research activities. We focus here on research activities carried
out by the e-Social Science Research Node at the University of Nottingham,2 and on the
experiences of social science researchers in the Equator Interdisciplinary Research
Collaboration,3 both of which are concerned to incorporate ethnography in the process of
technological innovation and design. Ethnography is an approach to social science research
that is already well established in IT research, particularly in CSCW. However, the
emergence of ubiquitous computing, which distributes collaboration across a burgeoning
array of small devices and online environments that exploit invisible sensing systems, raises
new challenges for the approach (Crabtree et al. 2006a).
The advent of ubiquitous computing allows a novel range of new experiences to be
constructed and deployed in the wild (e.g., Flintham et al. 2003, Crabtree et al. 2004. Benford
et al. 2004), and these experiences distribute collaboration across different applications and
devices, some online, some mobile, with each exploiting different mechanisms of
collaboration. Consequently, collaboration appears to have a much more ‘fragmented’
character than it does in traditional computing environments and understanding collaboration
in UbiComp environments requires reconciliation of those fragments; reconciliation of, for
example, what happens on the streets via GPS-enabled PDAs, with what happens online via
virtual models, with what ways collaboration is articulated between the two (e.g., via audio
on the one hand and text messages on the other).
Reconciliation requires ethnographers to supplement traditional resources external to the
digital setting of collaboration, resources generated by the ethnographer such as audio and
video recordings, with resources internal to the digital setting such as the text messages and
audio files generated by users in their digitally mediated collaborations. The need to develop
applications to support social science researchers in changing circumstances of design has
already been recognized by IT researchers. For example, Woodruff et al. (2002) and Brown et
al. (2003) have exploited the logs from electronic guidebooks and sensors to thicken
ethnographic and conversation analytic descriptions of collaboration. There is of course
nothing new in exploiting system logs to understand human-computer interaction. HCI
researchers have been exploiting system logs for decades and techniques for doing so are
documented in most good HCI textbooks. More recent efforts have sought to automate the
process (Ivory and Hearst 2001) and extend current approaches to support qualitative research
(Kort and de Poot 2005).4
Ubiquitous computing enables us to go beyond a concern with human-computer interaction
and the logging of machine states and events however, to record elements of collaboration
conducted and achieved through the use of ubiquitous applications as well. Thus, and for
example, audio and text messages may be recorded alongside machine states and systems
events. These digital records which detail interaction within a digital environment make a
range of media exploited in and effecting collaboration available as resources for the
ethnographer to replay and examine alongside external resources (Crabtree et al. 2006a). The
development of digital records, which ultimately combine internal and external resources, and
4 With the digital revolution social science researchers have also recognized the need to develop new software applications
to support the ethnographic exercise (e.g., Annotape, HyperResearch, Atlas, Qualrus, Anvil, and Elan). These, however,
tend to focus on manipulating audio and video recordings and on mining and/or coding textual data to support structured
forms of analysis. Furthermore, they are piecemeal in nature, offering some functionality here, some there, but no coherent
replay environment of broad utility. What is more, existing mechanisms for manipulating and marrying internal and external
resources together are limited and inflexible. In other words, and despite claims to the contrary, there are no existing
applications that permit the kinds of analysis that ethnographers involved in IT research require or which make the resources
exploited by users in future and emerging digital environments available to the broader social science community.
the core theme of ‘record and replay’ are the central focus of this paper. Below we consider
the case for developing digital records to support e-Research in general and ethnographic
studies of future and emerging technology in particular. We then provide two case studies,
one detailing the purchase of digital records to ethnographic study and the other detailing the
work involved in making digital records work, before moving on to consider the development
of a Replay Tool to support ethnographic study of collaboration in UbiComp environments.
2. The Case for Digital Records in Social Science Research
The advent of e-Social Science is underpinned by the development of a Grid infrastructure,
which is construed of as a distributed network of ‘virtual organizations’ that enable
computationally intensive tasks to be processed by a great many machines (Foster et al.
2001). The Grid vision is one well suited to the data intensive demands of the natural sciences
(see, for example, TeraGrid).5 However, the purchase of the vision is less clear with regard to
the social sciences. This is not due to the scientific status of the social sciences, nor to the
theoretical character of a great deal of research carried out by social scientists (which e-Social
Science may in due course change). Rather, the issue revolves around the nature of social
science data. Even large-scale research in the social sciences – such as survey-based research
– exploits computationally small datasets (Crabtree and Rouncefield 2005). While we must
be careful not to overstate the case, the demands of social science research are in significant
ways different to those of the natural sciences and this will affect the ways in which the Grid
vision is played out and realized in the social sciences. e-Social Science cannot be exclusively
predicated on intensive multi-computer processing however as large and small, ‘macro’ and
‘micro’, are not so much matters of data size but of analytic reasoning in the social sciences
(Benson and Hughes 1991, Hughes and Sharrock 1997).
Rather than focus on computationally demanding tasks in the first instance then, we think it
more useful to consider the future nature of social science records instead. We think it
particularly relevant to consider the ways in which computation might make new resources
available to social scientists through the recording of new forms of data and to consider the
ways in which data sets may be replayed, manipulated and transformed to support social
science analysis. It is here that computational complexity might enter into practice through
the design and development of new applications and services. We explore the potential for
record and replay in the context of ethnographic study, which we exploit in our own work to
evaluate the collaborative character of future and emerging technologies and to inform their
Since its migration from anthropology to domestic employment and the use of the approach
to study our own societies, ethnography has drawn on a wide range of resources to understand
social phenomena. For example, Robert Ezra Park, one of the most influential members of the
Chicago School of Sociology where the domestic use of ethnography was pioneered,
exploited ‘concentric zone maps’ along with fieldwork to chart the various social divisions
and boundaries shaping the urban environment (Park et al. 1925). The ethnographic tradition
draws on multiple resources to get the job of observing and analyzing social life from
‘within’ done, routinely exploiting biographical resources (e.g., Wieder and Zimmerman
1977), visual resources (e.g., Prosser 1998), technological resources (e.g., Heath and
Hindmarsh 2002), and a great many other resources besides. The point here is not to assemble
an exhaustive list of the resources used by ethnographers past and present – they are
contingent on the research setting and diverse - but to recognize that ethnography is done not
only through the immersion of a researcher in a setting but also through the use of material
resources as well. These resources may be external to the fieldwork setting, such as above
where diaries, or photography, or video cameras are introduced into the field, or they may be
internal and consist of such things as the objects of work, working documents, electronic
media, emails, etc.
External and internal resources already co-exist in the course of ethnographic research. Our
concern with resources here is novel, however. We do not simply wish to make resources that
are internal to digital settings of collaboration available as resources that permit closer
inspection and analysis of cooperative work, we also wish to marry them to external
resources and enable them to be replayed side-by-side to enable a more comprehensive
understanding of collaboration in future and emerging computing environments to emerge.
The development of digital records and tools for their use opens up this possibility and we
examine it in detail by examining the use of digital records generated within two different
digital environments (Cheverst et al. 2003a, Crabtree et al. 2004).
It is worth pointing out that we do not offer digital records as a solution to problems of
ethnographic inquiry or social science research more generally, only as a means of further
developing the ethnographic record or that collection of ‘stuff’ that the researcher brings back
from the field (Crabtree et al. 2000). Indeed, rather than being seen as something radical and
unprecedented or, conversely, as old hat, we suggest that digital records be seen as the natural
extension and evolution of the ethnographic record, where technologies of production have
progressed over time from paper and fieldnotes to incorporate a veritable host of new
computational media to record social life. As digital technology advances and becomes an
ever increasing feature of our everyday lives and relationships it makes historical and
contemporary sense to ‘move with the times’ (Crabtree et al. 2005) and to develop new
resources and tools that enable ethnographers and other qualitative researchers to investigate
interaction in the digital age. Having said that our own aim is not to extend such things as the
‘virtual ethnography’ programme (Hine 2000). No doubt digital records could be exploited
for such purposes, and we look forward to the use of e-Social Science applications by others
to do so, but our use of digital records simply offers a demonstration of the possibility of e-
Social Science to transcend existing resource boundaries and make tools supporting
collaboration in future and emerging computing environments into tools supporting social
science research as well. That, we believe, is what is novel about our own research, though
we acknowledge that it is being pursued by other researchers in the IT community as well
(e.g., Intille et al. 2003, Tapia et al. 2004). What is distinctive about our own research is the
effort to tie record and replay mechanisms to social science logics of inquiry and analysis
through design (Crabtree et al. 2006b).
Our focus on digital records is part of a much longer term and well-established course of
research that is concerned to understand the social character of technology. Two major
strands of social science research exist in this area. One is concerned to understand generic
social factors that ‘shape’ the uptake and use of technology (e.g., Virtual Society?).6 The
other is concerned to understand the situated character of settings of action and how the use
of technology therein is socially or collaboratively organized by participants or ‘users’ (e.g.,
COMIC).7 This latter strand of research is largely but not exclusively underpinned by an
ethnomethodological orientation to analysis (Garfinkel 1967, Suchman 1987) and is
interdisciplinary in nature. It involves close collaboration between social scientists and
computer scientists and has resulted in the incorporation of a social science perspective into
IT research, notably in the field of Computer Supported Cooperative Work where
ethnomethodologically-informed ethnography – or ethnography as we simply call it - is used
6 Virtual Society? http://virtualsociety.sbs.ox.ac.uk/
to inform and shape the design from within the research and development process (Crabtree
Our focus on digital records articulates recent attempts in this latter strand of research to
develop computer-based tools that better enable social science analysts to understand the
social organization of technology-in-use (Button 1992), in order to support the evaluation of
future and emerging technologies and to propel innovation and development (Crabtree 2004).
Below we present two case studies of the use of digital records in this distinctive research
context. The first demonstrates the utility of digital records to ethnographic studies of
technology-in-use. The second focuses on the work that it takes to make digital records that
contain multiple media into resources that actually support analysis in order to elaborate
requirements for design. It is important to stress that digital records do not stand alone but are
married to other resources that are external to the digital environment. In both of the cases
presented here the digital records gain their purchase as part of broader ethnographic inquiry
that exploits traditional techniques of immersion and direct observation along with the
gathering of data through fieldnotes, video recording, and more novel techniques such as
‘information probes’ (Rouncefield et al. 2003).
The first of the two cases focuses on the uses of digital records generated and recorded by a
messaging system designed to support collaboration between remote workers in a psychiatric
hostel. The second of the two cases is derived from a study of collaboration in a mobile
mixed reality game and focuses on the interplay between internal and external resources,
particularly on the marriage of text logs and audio files generated within a digital
environment and their relationship to video recordings external to the digital environment.
8 It is worth noting that the ethnomethodological orientation to ethnographic study is very different from that in mainstream
sociological or anthropological studies (see Lynch 2000, for example).
The work of using digital records generates requirements for record and replay applications
that promise to incorporate the rich mix of digital resources exploited by users in their
collaborative activities, and the case studies are followed by an account of an emerging
Replay Tool to support these requirements.
3. Case 1 - Working with Text Logs (SPAM)
In this section we describe a SMS Public Asynchronous Messenger (SPAM) system,
developed to support cooperation between psychiatric care staff working at two associated
sites in a small city in the North of England. The facilities support ex-psychiatric hospital
patients suffering a range of mental health problems. They comprise 1) a residential hostel
which is permanently staffed, and 2) a housing facility which supports semi-independent
living and is staffed during regular working hours. In the hostel, staff live and work alongside
residents and there is an office area for staff adjoining the hostel living room which is easily
available to residents. At the housing facility staff have a physically separate office and visit
residents in their semi-independent living flats, although residents regularly visit staff in this
office too (the setting is described in more detail in Cheverst et al. 2003a, Cheverst et al.
2003b, Rouncefield et al. 2003, and Graham et al. 2005). Here we focus on describing the
messaging system deployed at the setting, SPAM.
Initial requirements for the SPAM system (Figure 1) were elicited through ethnographic
study, informational probes, and design workshops, and it was deployed in a residential care
setting to enhance cooperation between the residential hostel and housing facility.
Figure 1. One of the SPAM displays.
Cooperation was supported by sending text messages between the SPAM units which were
located in the offices and provided shared displays (Figure 2). The SPAM system was
designed to run an SMS messaging application, allowing staff in the two sites to
communicate easily by composing messages using an on screen keyboard displayed on a
touch sensitive screen. When a SPAM unit receives messages they are displayed on the
screen until deleted by a member of staff. Staff can also use their own mobile phones in order
to send text messages to the SPAM displays when they are out of the office and to receive
messages originating from a SPAM display.
Figure 2. Architecture of the SPAM system.
All messages are routed through two SPAM units, making logging of text exchanges
possible. A typical use scenario is illustrated in Figure 2 by SMS Message 1 - i.e., the
message originating from a mobile phone which is delivered to the permanently staffed hostel
(Location B) and the transmission of a ‘message read’ acknowledgement is triggered by a
member of staff reading the message. Message forwarding is performed by the system if a
message is sent to the semi-independent living accommodation (Location A) at a time when
no member of staff is providing cover (denoted by AWAY STATE). In this case, the message
(Message 2) is automatically forwarded to the display of the other hostel.
The SPAM system was deployed in 2002 and over time it has come to be viewed by staff as
an additional tool for communication which is capable of supporting them in their everyday
work. In short, and as Sacks (1992) suggests, the SPAM system has been “made at home in a
world that has whatever organisation it already has”. In texting to a shared display, users of
the SPAM system can make available to other staff their location, plans and activities, and
thereby draw upon and reflect on certain social, temporal, spatial and relational aspects of
everyday life that are essential to cooperative work (Hughes et al. 1997). More specifically,
the SPAM system ‘affords knowledge’ (Anderson and Sharrock 1993) between care workers,
displaying work in the ‘here and now’ and making it available for articulation and
coordination across sites (Schmidt and Bannon 1992). Texts to the SPAM system become
both the focus of work and a visible record of work that has been done, work that has been
put on hold, work that remains to be done, and so on. By effectively embedding messages in
the everyday fabric of the workplace, in the office furniture in effect, by putting the work on
display so that others may be aware of it, these textual representations make everyday work
‘visible’ so that it can be ‘taken note of’, ‘reviewed’, ‘queried’ and in other ways be made
accountable by and for others involved in the work, thus affording what Erickson and
Kellogg call ‘social translucence’ (Erickson and Kellogg 2000).
However, the digital record of collaboration conducted via the SPAM system is fragmented
by space and time. The digital record is temporally fragmented due to the system being
asynchronous, and spatially fragmented due to the two SPAM terminals operating across two
sites. The SPAM system exemplifies a series of problems to do not only with the recording of
collaboration but also with the analysis of the digital record of collaboration. These problems
are worked through in the next sections and possible approaches to reconciling the fragments
of distributed collaboration across the two sites are considered.
3.1 Data Issues
We have previously outlined a range of factors in psychiatric care settings that render our
usual ethnographic data collection techniques problematic (Cheverst 2003a). Similarly, we
have outlined how we have sought to supplement our understanding of settings ‘from within’
by adapting Cultural Probes (Gaver 1997) to gather data about participants’ daily lives, to
elaborate the needs of users, and to sensitise parties involved in design to the settings within
which new technology will be embedded (Rouncefield et al. 2003). The SPAM system also
acts as a probe – i.e., as a means of gathering data – functioning in a great many respects as a
‘technology probe’ (Hutchinson et al. 2003). The SPAM system has non-intrusive recording
functionality enabling monitoring of the ongoing use of the technology and the collection of
usage information in a real world, real time setting. The textual records generated provided us
with a complementary source of information that could be interrogated to provide some
measure and assessment of the functional value of the system in the context of its day-to-day
The SPAM machines perform recording by appending messages to a plain text file. Figure 3a
shows a sample of the log file entries generated by the SPAM system for a message sent to
Location A from Location B. Figure 3b shows a sample of the log entries generated at
Location A when this message arrives. The result is a mixture of debug output from
communication with the GSM terminal, text content, and higher-level elements indicating
that a message has been sent, received etc. In his classic account of sociological research
methodology Documents of Life, Plummer (1983) carefully sets out how a range of
commonplace documents such as diaries and letters can be interrogated and analysed in order
to afford various insights into the character of social life. Without buying into the narrative
turn that often accompanies such insights, we want to point to the growing prevalence of
digital documents of life as unavoidable and interesting features of everyday life and work.
The emergent plain text file generated by the SPAM system provides us with a concrete
Figure 3a. Log of message sent from Location B to Location A.
Figure 3b. Log showing message received at Location A.
The difficulties of using and analyzing textual elements of digital records derives at least in
part from the strange or at least ambiguous place that the technology and its recording
function occupies – both in terms of social research methodology and in the setting itself.
Embedded in the technology the recording function performs various kinds of work that goes
beyond the actual work of text messaging itself and can impact on the process of using the
digital record for understanding the setting and its activities. Working with text logs, using
them for both social analysis and design work, involves working with, working through and
resolving a number of quite complex technical and analytic issues; difficulties that generally
go beyond those usually encountered with other fieldwork technologies. Of course, issues
such as the need to set internal and external records against each other is hardly novel in the
general run of social science research but, in the specific instance of digital records - and the
kinds of records we are looking at in future and emerging computing environments - the
difficulties encountered are of an entirely different order to those that usually confront social
In this and other instances (e.g., Crabtree and Rouncefield 2005, Crabtree et al. 2006a,
Crabtree et al. 2006b) considerable work has to be done with the digital record to make it
available as a resource for analysis: extracting system state messages, tracking messages
from one site to another, examining when messages get written, get sent and get read, for
example. A key technical issue revolves around what information to record in the first place.
While it is sometimes obvious from the outset which information will give the best clues
about use, it may not always be apparent until after thorough analysis of a digital record
exactly what additional information it might have been useful to collect, however.
Additionally there are limits on the amount of information that can be collected, so it is
necessary to strike a balance between what can be collected, what it is essential to collect, and
what it is possible to store. Of course, as cautious social scientists somewhat obsessed with
the need for data, our general policy is always to collect more information than may appear
necessary, since what is important, relevant and interesting is not always immediately
When we attempted to analyse the SPAM records in order to examine the day-to-day
dialogue taking place between staff, we found handling the recording function an unexpected
challenge. After attempting various means to parse the record in different ways, programs
were written to extract messages sent and received by the SPAM system. These were then
placed in separate text files, which separated and formatted the entries. Initially it was very
hard to follow the chronological order of dialogues using two separate files for messages sent
and received, so the analysis program was modified to output to a single file. Unfortunately,
however, we found that only the time and date of messages received had been recorded, not
the time and date that messages were sent (Figure 4).
Figure 4. Parsed log showing messages sent from and received at Location B.
The SPAM system did provide an acknowledgement reply SMS message when a message
had been read, which meant that the next entry in the record gave a good approximation of
when the previous message had been sent. Obviously this was not ideal and made analysis of
the record more difficult, as the acknowledgement entries made it harder to see the actual
messages being sent and received (and should ideally be filtered out). Our solution to this
problem was to modify the analysis program to make the acknowledgement entries much
smaller (so they only take up a single line), and to highlight by hand the messages sent and
received using different coloured marker pens. Additionally we performed ‘find and replace’
to add names to known mobile phone numbers. All in all we found that exploiting the
recording functions of the SPAM system raised some non-trivial challenges and appropriate
support for recording events needs to be considered at design time given the potential
implications that appropriate support can have. One requirement that is perhaps more peculiar
to UbiComp systems (given the potential range and number of sources of recorded
information) is the need to consider the design of appropriate tools to support the integration
of various records and the need to support human augmentation (e.g., annotation) of these
logs. We have found this latter requirement to be a key requirement for analysing usage
patterns from SPAM.
3.2 Analytic Issues
One of the core problems of working with the digital records is the indexical nature of
language (Garfinkel 1967). This is a ubiquitous problem and means that our understanding of
the contents of digital records relies on an understanding of the setting and collaborative work
that the technology is embedded in. It is not apparent at a glance just what “liv 1 ch 0
cheriou” means, for example, though immersion in the setting of work informs us that it
refers to the result of a football match. Thus, while our use of the digital record was and is
intended to support ethnographic analysis, an understanding of the setting generated through
field work and (in this case) the deployment of probes was an essential ingredient in the effort
to make sense of the data recorded in the digital environment and highlights the mutually
supportive nature of digital records and ethnographic methods.
Furthermore, by exploiting grounded analysis techniques (Strauss 1987), some success has
been achieved in eliciting categories and general themes from the textual record that describe
SPAM usage patterns, the results of which are represented in rudimentary software tools such
as spreadsheets (Figure 5a and Figure 5b).
Figure 5a. Spreadsheet used to support analysis of SPAM messages into categories.
One challenge with such analyses is to uncover the subtle character of collaboration as it
plays out in the setting when the digital record essentially distances the analyst from that
setting. Although important sequences of interaction can still be identified through the
technology (as can be seen above), these sequences are ‘thin’ and lack the ‘thickness’ typical
of ethnographic description (Crabtree 2003).
Figure 5b. Spreadsheet visual tools representing general patterns of SPAM message use.
In addition, while general descriptive categories and theme sets (e.g., Figure 6a) and their
visualization (e.g., Figure 5b and Figure 6b) are a useful means of identifying the different
kinds of interaction that take place in a setting and their frequency, they represent a view on
the use of the SPAM system that is frozen in time and space and is both non-indexical and
deceptively definite. Such techniques, tools and representations although not wholly
constitutive of analytical reasoning itself, form an important part of any toolkit supporting
analytical reasoning by enabling categorical views on digital records to be generated.
Figure 6a. Description of categories and sub-categories emerging from analysis of SPAM logs using
grounded analysis techniques.
Figure 6b. Visual representing frequency of and relationship among categories from analysis of SPAM
logs using grounded analysis techniques.
While we acknowledge these limitations, findings of a real world, real time nature have
emerged from the analysis of the textual record through the use of these tools in combination
with fieldwork observation. For instance, an outsider, with little understanding and knowledge
of such settings, might assume SPAM-based communication to revolve around information
exchange in safety critical and emergency situations, and that asynchronous text messaging
must be inferior to technology supporting synchronous communication in such situations. Yet
analysis of a sample of the text logs (and indeed development of categorical views of the
digital record) showed that most communication - 31% of messages sent - was prosaically
social and involved such mundane matters as exchanging football scores, telling jokes,
making fun of each other. SPAM also had a pivotal role in supporting other communication
channels in the setting (and, indeed, this was a key reason for its introduction): 26% of
messages sent supported ongoing work interaction, including switching to other channels
such as the phone or fax machine, and additional analyses also show the importance of SPAM
for mediating awareness (see below). These findings do not trivialise the use of the
technology but point to how users actually appropriated it for affective, non-functional
purposes that are, nonetheless, essential to maintaining contact in a work setting that is
distributed, occasionally safety critical, and often painfully mundane.
Despite the difficulties associated with the indexical nature of digital records and the hazards
of exploiting general descriptive categories and theme sets, we believe that when coupled
with external resources, interesting and important material may be produced that helps us
understand the recorded data in terms of ‘everyday occurrences’ and as constituent features of
ordinary workaday activities. In this case the external resources, such as fieldnotes, were not
placed alongside the digital record to thicken up descriptions of discrete sequences of
interaction, but instead informed the identification of collaborative characteristics of
technology use. This is not to say that tools supporting real-time reconciliation of internal and
external resources would not be useful and enrich the resulting description. It is simply the
case that such tools were not available to us and being able to map messages onto other
events in the setting such as observations of staff behaviour would have been very be useful.
Nevertheless, certain events surrounding the digital record were available to us: Saturdays
tended to mark a spike in social interaction around football scores because it is well-known
that many football matches are scheduled on that day, for example. Nevertheless, it remains
the case that being able to explore the detail of other events, particular to the setting, which
surround message exchange during the course of work and subsequent analysis would have
given us additional insights into SPAM use.
Digital records have been important in showing how the texting technology found a place
within and was responsive to the working sensibility of a setting, how the technology was
‘made at home’ at the sites it inhabited, and how it came to fit into and resonate with
everyday work. At present, analysis suggests that the texting facility plays some part in
promoting various forms of awareness (e.g., “Has fax, email got through? Has X left yet?”);
forms of coordination (e.g., “I keep ringing and nobody answers? Can you ring me please”;
“Pizza & and chips ready come on in ?” “Jane can you ask jo to ring me when she comes in
about the swop”); various forms of monitoring and tracking (e.g., “What shift is peter doing
tomorrow and where”; ) as well as general queries (e.g., “Which keys should we hand
over?”). The growing use of SPAM to conduct mundane activities suggests the technology is
slowly but surely becoming organizationally embedded in the day-to-day work of the
residential care setting. The digital record suggests that texting in the hostel allows workers to
negotiate their availability and maintain their connection with the rest of the staff. Knowing
who is around, what people are doing at weekends or during sleepovers at the main hostel, for
example, enables workers to establish and project a range of possible collaborations, much as
door displays in a different setting allow people to project appropriate courses of action in
response to messages left by users (Cheverst 2003b). Texting, in other words, enables users to
plan joint activities as much as it enables their coordination.
The organizational character of texting consists of an explicit construction and sharing of
context in order to support (or potentially support) collaboration with others. What becomes
obvious in reading the digital record is the flexibility of text messaging in terms of supporting
the everyday work of the care hostel. The expressive character of texting is also noteworthy.
Even without the addition of emoticons, our users routinely employ texting for affective
communication about work, work crises, jokes and general social banter. As Nardi et al.
(2000) put it,
It is interesting that a lightweight technology consisting of no more than typing text into a window
succeeds in providing enough context to make a variety of social exchanges vivid, pleasurable, capable
of conveying humour and emotional nuance.
For the social scientists on the project, the record and reuse facility of the text logs provided a
valued and worthwhile resource that supplemented existing social research techniques and
facilitated our understanding of social action from members’ point of view as their work
unfolded. In other words, the logs provide a record of social action in real time (Sharrock and
Button 1991). People cannot know how their activities will turn out – whatever their
intentions and best efforts, the workaday world is replete with contingencies – and these
happen in real time. Consequently getting a better understanding of members’ point of view
requires the examination of the organisation of social action over its course. Members’ point
of view is temporal and resides in unfolding, ongoing action. The digital record presents an
opportunity to understand something of the real time context of action as it allows us to
examine the ‘social actor’ as a practical doer of concerted (collaborative) ongoing activities
and, in that respect, to make visible the kinds of things that members need to ‘get done’.
Thus, when combined with external resources, the digital record reflects the concerted
decisions members make as an embedded feature of carrying out a course of action.
Concerted judgements and judgement calls as to ‘Where are we now?’ ‘How much have we
done?’ ‘Is this course of action working out as we anticipated or do we need to adjust the
prepared course?’ ‘How much more is there left to do?’ ‘How can we get from doing what we
are doing now to doing what we need to do next?’ ‘What do we need to do next, exactly?’
And so on. To the extent to which digital records reflect and document the practical workaday
concerns of members we have found them invaluable. This is not to suggest that analysing the
record is easy, for as already noted, the data it contains is indexical to the activities that
generated it. Knowledge of those activities developed through fieldwork techniques is
brought to bear on analysis of the digital record and make it meaningful. In other words, the
digital record depends for its practical adequacy – it’s salience to the analysis of cooperative
work and systems design in this case - on knowledge of the activities in which the technology
is embedded and used. That knowledge is used to make sense of the digital record or to
interpret it where structured forms of analysis are concerned (Garfinkel 1967, Crabtree et al.
2000). While the digital record acts as an important resource for understanding and
explicating features of everyday action in digital settings, its sense and reference or meaning
is dependent both on prior and ongoing understandings of the setting developed through other
means of inquiry. It is in this respect that digital records detailing endogenous features of
social action – i.e., features that are internal to and constitutive of action itself - and
ethnographic methods are said to be mutually supportive.9
4. Case 2 - Making System Logs Work (Uncle Roy)
The mutually supportive character of digital records and ethnographic methods means that
augmenting future and emerging computing environments to enable ethnographers to exploit
resources internal to collaboration can add real and tangible value to analysis (Crabtree et al.
2006a). Exploiting such resources takes work, however, and in this section we wish to
explicate in greater detail the work involved in making digital records work. The record in
this case is taken from a mixed reality game called Uncle Roy All Around You.10 While it may
appear trivial to many hard core e-Science advocates, gaming is a burgeoning area for
developing future and emerging technologies in a safe and engaging context that involves a
large number of users in real world, real time circumstances (see the special issue of JCSCW
on Leisure Technologies, forthcoming).
9 We are not the first, of course, to suggest that various resources internal to work may be mutually supportive
of ethnographic methods. As one of the reviewers of this paper reminds us, “several researchers who have
combined ethnography with more micro level (interaction) analyses have noted the same (e.g., Suchman, Trigg,
Blomberg et al., Jordan, Henderson, Ruhleder et al. and Heath, Luff, Jirotka et al.) … Some of this previous
work … deserves to be mentioned.”
To play the game, street players must navigate the streets of a city and locate Uncle Roy’s
office. To do this they use a handheld computer or PDA that provides a map view of the city
streets. The players receive automatically generated clues providing them with directions to
particular places in the city and further clues are provided when they declare their current
position. Street players also receive text messages from online players, who can track a street
player’s movements through a virtual model of the actual city streets. Street players can
respond to online player messages by recording a short (7 second) audio message. Working
via text and audio messages online players and street players collaborate to find a postcard.
Once found, online players are provided with information that they use to guide street players
to Uncle Roy’s office (see Benford et al. 2006 for a full description of the game).
The technology employed in the game captured the clues sent to street players, the text
messages sent by online players, and the audio messages sent by street players, and combined
them together in a single record with associated audio files. An ethnographer also
accompanied several street players as they made their way around the city (Manchester in this
case) and recorded their interactions on video. In order to develop an understanding of the
social or collaborative organization of the game it was necessary to combine these internal
and external resources to furnish coherent sequences of collaboration for analysis and it is the
work of combining resources that we examine below. Although the raw record captures the
features of collaboration described above, those features are not readily accessible or
amenable to ethnographic description and analysis, as can be seen in Figure 7. The record
consists of wide range of information, including temporal information (in terms of system
time but not ordinary clock time), spatial information (i.e., player movements and
declarations), system events (audio events, chat events and clues), and collaboration events
(text messages and audio file identity). In order to turn the record into a usable resource for
purposes of ethnographic description and analysis it is necessary to clean it up so that salient
features can be identified.
Figure 7. The raw digital record.
In this case ‘salient features’ consist of those features of the record that enable the
ethnographer to understand collaboration between online and street players. Salience is also
determined by external resources, in this case the video of collaboration that the ethnographer
has captured on the streets. Cleaning the record consists of identifying parts that temporally
‘sit alongside’ external resources and requires non-relevant information to be stripped out of
the record, as in Figure 8 where a logged sequence that covers the collaboration captured in a
particular video sequence is presented.
Figure 8. Cleaning the record.
The notion of sequence is essential to cleaning the record and it refers here to interactional
sequence – i.e., the order of interaction or collaboration captured on video and in the record.
The work of cleaning the record continues with an eye towards identifying and extracting
only those features that relate to the interactional sequence documented on the videotape. As
can be seen in Figure 8, multiple interactional sequences or ‘conversational threads’ populate
the record. Not all of those conversational threads are relevant to the interaction captured on
video, however, and so further extraction is required to identify salient threads. In this case,
what is of interest are the threads that pertain to Patrick (a player whose collaborations were
recorded on video), so cleaning up the record involves extracting conversational threads
between particular participants from the flow of overlapping threads between the multiple
participants represented in the record. The cleaned up record actually looks like this then:
Figure 9. The cleaned log.
Cleaning up the record relies on making determinations of relevance – at its simplest level,
determining what’s ‘noise’ and what’s not (a judgement that is relative to the kind of
description and analysis to be performed). Such determinations rely on the use of other
materials: not only on video but also, in this case, on the audio files generated by Patrick in
his collaborations with online players Venom, Nicole and Dave. In order to understand the
salience of the audio files to interaction it is necessary to transcribe them, as in Figure 10,11
and then synchronize them with the text content of the record as in Figure 11.
Figure 10. The transcribed contents of the audio files.
Figure 11. Synchronizing text messages and audio files.
The job of synchronization continues by transcribing the contents of video and marrying it to
the audio file transcripts and text contained in the digital record, as in Figure 12.
11 There are of course different methods of transcription and the more exacting of these methods, such as Conversation
Analysis (Sacks et al. 1974), might benefit in situations such as this by converting system time that is recorded alongside
logged events into ordinary time to support fine-grained transcription.
Figure 12. Synchronizing video, audio and log content.
It is at this point of synchronization that the use of digital records becomes analytically
interesting and troublesome.12 Accordingly, when we turn to the work of synchronization we
12 This is not, as one reviewer suggests, because the real problem of e-Social Science is “data mining” or “knowledge
extraction”, which requires “ontologies to provide context and improve interpretation”. The imposition of ontologies on
social science research is not, in and of itself, going to improve anything and we would suggest that e-Social Science
can see that the intertwining of text content, transcribed audio content and transcribed video
content brings the record to life in that the contents of the record start to assume some kind of
recognizable sense, compare Figure 7 with Figure 12, for example. Unlike the representation
in Figure 7, that in Figure 12 allows us to see that greetings and introductions are made, that
collaboration sometimes goes no further, that the local knowledge of passing members of the
public is drawn upon by street players to navigate the city streets, that instructions are issued
by online players orienting street players to specific features of the streets (phone boxes), that
collaborations ensue and are directed towards finding postcards for the online players, that
such collaborations lead street players off track and require the intervention of Uncle Roy,
and so on.
We can also see that what online players hear from street players as conveyed to them by the
contents of audio files does not represent all of what is said by street players. The bold typing
in Figure 9 represents the contents of audio files and from this we can see that such things as
requests for directions or for particular actions to be done (such as texting the name of
someone from your past who never leaves you) are not heard. This may impact upon
interaction and in part account for the breakdown of interaction between players (which was
not an uncommon event). We can also see that not all ‘utterances’ are treated as relevant by
street players to situated action on the streets. For example, Patrick’s actions do not turn upon
Nicole’s utterances 24, 31, 38, 42. This is not to say that Patrick does not see them but that
they are not relevant to or responded to in Patrick’s ongoing interactions ‘here and now’ on
the streets. Neither is it to say that these utterances have no part in playing the game – while
Patrick does not respond to them, some of them they are important to gameplay, with
utterances 38 and 42 triggering a response from Uncle Roy where the information needed to
developers will do better to proceed by examining the troubles that naturally inhabit different kinds of social science
research, not all of which exploit “ontologies” (Garfinkel 1967).
guide Patrick to Uncle Roy’s office is furnished to Nicole. Nevertheless, interaction on the
streets is not driven by each and every utterance made by players online but is instead shaped
by the exigencies of the situation on the ground, and it is in this respect that the synchronized
record is troublesome.
Recording and organizing contents in terms of system time, the digital record offers a
seductive representation of the sequential order of collaboration. System time is misleading
however, as it fails to represent the sequential order of interaction of which collaboration
consists. It is critical to understanding collaboration in UbiComp environments to understand
when recorded events that are part and parcel of collaboration – text and audio messages in
this case - enter the interactional situation and not only the time when they are recorded by
the system. System time and interactional time are different: one is driven by the measurable
linear progression of some standard unit whereas the other is driven by the exigencies of
practical action and this has ramifications for the way in which interaction/collaboration is
represented (described) and understood (analyzed).
In order to appreciate the impact of describing the interactional order of recorded events, in
contrast to the recorded order of interactional events as represented in Figure 12, we might
compare the two. Figure 13 shows the relative positions of recorded events when logged and
organized by the time in comparison to when those same events observably and reportably
entered the interactional situation as articulated by the street player, witnessed by the
ethnographer, and documented on the videotape. The re-ordering of recorded events to reflect
interactional order has a profound effect on the shape of the record, with online players
utterances finding a new place, sense and purchase that articulates the interactional situation.
For example, the highlighted utterances in Figure 13 show the interactional movement of log
entries from linear entries to interactional entries where they initiate and respond to specific
actions. This representation is not to suggest that events in the digital environment were
recorded incorrectly, but that they did not become accountable features of interaction until
their occurrence in the places they occupy in the interactional order.
Figure 13. Comparison of logged order and interactional order: where logged events actually enter
The reordering of recorded events is driven by examining the video and placing them in
relation to accountable actions: to what street players say and do. Placement is shaped in
particular by the retrospective-prospective sense of meaning that is conveyed by an utterance
or action, which suggests that establishing what any prior utterance or action means depends
on considering what follows it (Garfinkel 1967). Thus, and for example, the re-placement of
Dave’s utterance “u need the phonebox on portland street by the tower” from number 25 in
the recorded order of events in Figure 12 to number 29 in the interactional order in Figure 13
is driven by Patrick’s account which accompanies his looking at the PDA, “Right, and
apparently there’s another one. There’s a phone box – Dave’s telling me there’s a phone box
over by Portland Tower.” This is not to say that Patrick has not seen the message before -
maybe he did read it when it was recorded by the system. Rather, it is to draw attention to the
observability of action and that it is only now, in relation to this action and those that follow,
that the utterance enters interaction and becomes an accountable feature of collaboration
between online player and street player.
There are, of course, consequences to reordering digital records, not least that it fractures the
recorded timeline between events. Thus, and for example, utterances 4, 19, 29, 33, 35, 37, 38,
and 39 in the interactional order of events are not only replaced but replaced in a different
order to which the system recorded them. This has serious implications for the design of tools
to support the enterprise and it is towards unpacking and addressing these issues that we now
5. Developing Computer Support
Consideration of the work involved in making digital records work has drawn attention to a
number of broad requirements for the development of computer support. These include:
Setting internal and external resources alongside each other.
Selecting and extracting relevant parts of digital records.
Synchronizing relevant parts of the record temporally with external resources.
Stripping out non-relevant features of digital records, such as conversational threads
that are not covered by external resources.
Transcribing audio and video content and marrying it to textual content contained in
Reordering digital records to represent the interactional order of events rather than the
recorded order of interaction, including changing the temporal order of events.
Below we present a Replay Tool that is being developed to support ethnographic description
and analysis of collaboration in UbiComp environments, and we outline existing functionality
and future challenges.
5.1 Replay Tool
Replay Tool consists of a four discrete components, each of which assists the user with the
replay and analysis process. First, information in the digital record is parsed and stored in a
database to allow detailed querying of the data it contains. This parsing process is, at present,
tailored to particular log file formats, though work is being carried out to enable the tool to
parse various formats of log files after the user provides a skeleton description. As part of the
current parsing process, recorded data is rendered into human readable form by, for example,
replacing system codes with real user names where available and, similarly, labelling
utterances to indicate who they were from and who they were directed to. Each log file event
is allocated a point on a timeline derived from when the event was recorded.
The second component of the tool allows the user to manage the data repository and view
recorded information in various ways, e.g., temporally or spatially as in Figure 14 below.
This enables an overall picture of the available data to be built up. These visualizations of the
raw data present a much more accessible format to the ethnographer as compared to the raw
record (Figure 7): temporal views reveal thick and thin areas of data availability at-a-glance;
data for particular users at particular times can be viewed on the timeline; and spatial views
enable the ethnographer to see a user’s journey through the physical environment (and in the
future will index the data associated with individual users to allow the ethnographer to access
all the data that is recorded and available at any point in a users journey through a place).
Ultimately, these visualizations enable the ethnographer to concentrate on the data itself
rather than the onerous task of teasing data out from the raw record.
Figure 14. Visualizing the log data: overview of data types over time and map of user movements in the
The third component of the suite is a tool to aid initial synchronization of data sources. Any
temporal section of the record can be extracted by setting time limits (start and end times).
This extracted section is then saved as an intermediate log file in human readable format.
This tool also draws in external media referenced in the record – so, for example, audio files
referenced in the Uncle Roy record are made available using hyperlink-like log entries. Each
log event in this file has a corresponding timestamp. Also in this component, external media
files, such as video files, can be imported into the project and allocated a start time to marry
internal and external resources that together articulate the collaborative character of particular
sequences of action.
The final component of the tool is the Replay Tool viewer itself, which allows the
synchronized replay of multiple data sources using information from the previous component
of the software. Development so far has concentrated on the use of log files and video files
Figure 15. The Replay Tool after a log file section and a video have been imported.
These media sources can then be replayed and navigated concurrently, with the appropriate
line of the digital record being highlighted to correspond with its place in the video resource.
The resources can be navigated using standard video transit buttons, or by clicking on lines in
the log window, which takes the ethnographer to ‘that’ point in the video. The two media
types are automatically shuttled to the correct place when the viewing time of one of the
resources is changed.
Once the media have been imported into Replay Tool, the user is able to annotate these
resources. Annotations currently exist at points in time stored in a relational database, and so
once made can be replayed against any temporal media resource. These annotations are
flexible in nature and can include transcriptions of audio files or links to external resources
such as digital images, documents or any other digital media type. Annotation functionality
enables the ethnographer to ‘trawl’ through the digital record, making rough notes,
identifying points of interest, marking out topics, etc., and to exploit annotations as an index
into the record, thereby supporting the descriptive and analytic process where representations
are worked up over the course time and where the work of description moves from initial
inspection to the shaping of coherent representations for analysis (Crabtree 2003, Crabtree
and Rouncefield 2005)
Figure 16. Annotated resources appear interlaced with the log data and as a dynamic index of
Once imported into Replay Tool, there is some functionality for the user to reorder or remove
log entries and annotations. At present this is done by mimicking the cut and paste aspects of
word processors for simplicity and familiarity of use. Nonetheless, this functionality enables
the ethnographer to clean up the digital record, to delete non-relevant features and reorder the
record to reflect the interactional order of events (Figure 17).
Figure 17. Working up a representation of the interactional order of events using annotations, deletions
and reordering recorded data.
The result of this process of marrying internal and external resources together and of
annotating, cleaning and reordering the digital record produces a result that is the same as the
interactional representation provided in Figure 13, and which also incorporates a fine-grained
temporal order at little overhead to the ethnographer: only minor annotation is required rather
than manual manipulation and annotation of the whole digital record or section of it and
therein lies the purchase of the Replay Tool. Prior to development of Replay Tool, working
with digital records was in many respects manual work, which relied on the ad hoc use of
existing and generic word processing software to extract and merge salient features. Replay
Tool enables the process of description and representation by supporting data visualization,
synchronization, annotation, and the reordering of log entries. That is not to say that Replay
Tool provides a solution to the problems of ethnographic study of collaboration in UbiComp
environments. Clearly it goes some way towards this but a number of complex challenges are
still outstanding and need to be addressed in future work.
5.2 Future Work
We have already mentioned the indexing of data to spatial visualizations of data and this
represents a major area of development to explored in the future. While there are, no doubt, a
great many challenges here we think that they revolve around two major axes: ecological
representations, where representations of physical and virtual spaces become resources for
organizing and accessing recorded data, and egological mechanisms of recording, where the
ethnographer’s location and proximity to users becomes a resource for recording data:
mechanisms which log spatial and temporal coordinates and which exploit wireless
technologies to capture computer-mediated collaborations alongside audio and video
recordings, for example (see Barkhuus et al. 2005 and Crabtree et al. 2006a for early work in
this area).While work on each of these areas of research is currently underway, three other
matters of pressing concern - synchronization, annotations, and representational structures –
require that we address them more immediately.
A core issue of synchronization raised in this paper is the ability to handle the difference
between interaction time and system time – there is a clear and necessary distinction between
when events are recorded by the system and when they actually enter collaboration. This is a
ubiquitous feature of interaction: think of mobile phones and text or audio messages, for
example – when a message is sent and recorded by the system is not necessarily when it is
acted upon by the recipient, which might not occur until considerable time afterwards, if at
all. Future iterations of Replay Tool will exploit a more sophisticated model of events in the
digital record based on an observation-oriented paradigm, to handle the apparent discrepancy
between system time and interaction time. This model assumes that any event only becomes
known when it is observed, and the ‘same’ event may be observed by different observers and
from different perspectives. While each observation can be characterized by a particular
period of time, these need not be the same for different observations of the same event
(consider for example the delay between seeing and hearing lightening). Therefore, it is
sensible to model the event description (event) and time parameters (observation) separately.
Using this model, one event can have multiple observations in the digital record, each with a
different time, e.g., one from the system recording point of view and one from the perspective
of an analyst examining real world arrangements of collaboration. The number of possible
observations is unlimited.
Figure 18. An overview of the proposed event-observation architecture.
In many cases a particular observer (such as the system or the ethnographer) will observe
many events, and the timings of its observations will have a consistency within that particular
perspective. To preserve this consistency within the digital record we can define multiple
timelines, each corresponding to a particular perspective or observer, such as a timeline based
on a particular system clock on a server or PDA, or from a video camera. Each observation
Description of event
A timing perspective
Base timeline for
Codes, free text or
media related to this
may then be assigned to a particular timeline. Using these features it is therefore possible to
maintain multiple orderings of events representing different perspectives on the data. It is
also possible to relate timelines to one another, for example by a list of ‘equivalent’ or
corresponding moments on the two timelines, and so allow observations to be mapped from
one timeline to another, at least to the extent that the corresponding perspectives are
compatible. For example, simple direct mapping could be used to ‘prime’ the interaction
timeline with initial proposed observations of system-recorded events, which can then be
worked up and refined by the ethnographer alongside the video record while still retaining
their relationships to the common events and the system’s own perspective. This new
architecture should aid the understanding and interpretation of the reordering of events
illustrated in Figure 13.
Less complex but just as important is the need to extend annotation functionality. Annotations
as they stand are free-text user entries stored in the database at a point in time. This makes
them tied to the temporal dimension, which may not be appropriate in all circumstances.
Consider, for example, when the ethnographer wishes to annotate an external resource such as
a particular physical document – this annotation is essentially independent of the timeline and
suggests that maintaining the link between the object of the annotation and the annotation
itself is what is important. Currently, audio transcriptions are still tied to a common timeline,
for example (the ethnographer’s interactional timeline), but it is more sensible to link them to
the audio file from which they were derived, and then tie the audio file to the interactional
timeline. In turn, this kind of mechanism may enable ethnographers to address another
ubiquitous problem encountered in using system recordings alongside external resources,
namely, temporal slippage where video recordings and internal system recordings slip out of
synchronicity over time. Providing mechanisms that allow ethnographers to construct and
maintain the link between data objects, and to manually configure and reconfigure their
temporal relationships, will enable them to address this problem.
Additional development will also be shaped by the ability to create and apply representational
structures or ‘coding schemes’ or ontologies to digital records. The current incarnation of
Replay Tool only supports free-text, but many ethnographers exploit representational
frameworks to organize data and extract findings. There is a clear need to support the
structured analysis of digital records and work is ongoing to identify mechanisms to support
this. While we have yet to establish just what such mechanisms might ‘look like’ concretely,
we recognize (unlike many existing applications) not only the key role of instructions to
coding but also the inevitability of ad hoc considerations to their application. As Garfinkel
(1967) puts it,
To treat instructions as though ad hoc features in their use were a nuisance, or to treat their presence as
grounds for complaint about the incompleteness of instructions, is very much like complaining that if the
walls of a building were only gotten out of the way one could see better what was keeping the roof up.
This observation draws our attention to the need to support the ‘art and craft’ of coding work
(to look at what keeps the roof up as it were) rather than devise support mechanisms that
blindly apply pre-formulated schema to digital records. So the requirement here is to attend
closely and carefully to ways in which ethnographers work with codes, as we have sought to
develop support for those that don’t, and to develop mechanisms that support the essentially
ad hoc and skilful nature of the enterprise. This course of research is currently being pursued
by colleagues in the DReSS research project and results will be reported in due course.
The work reported here has been conducted under the auspices of the UK e-Social Science
initiative and represents a departure from existing models of e-Research, which essentially
place emphasis on Grid-enabled super computing. While well suited to the needs of natural
science, which this model emerged from, it is less clear as to its salience to social science
research where scale is a matter of analytic orientation rather than data size and
computational complexity. Consequently, we have pursued a different path towards e-
Research where emphasis is placed on developing new forms of digital record that marry
existing resources with novel ones to add value to the enterprise. We have explored the initial
development of digital records and replay tools that enable ethnographers to handle the
fragmented character of interaction in ubiquitous computing environments. In such settings
collaboration is not only distributed but mediated by a burgeoning array of small devices,
online environments and invisible sensing networks. The challenge for ethnography in this
context is to reconcile the ‘fragments’ of interaction that are distributed across the various
parts of ubiquitous environments – to reconcile what happens on the streets via GPS-enabled
PDAs, with what happens online via virtual models, with what ways collaboration is
articulated between the two (e.g., via audio on the one hand and text messages on the other).
The development of digital records enables the capture and analysis of these fragments and
the value they might add to ethnographic research has been explored through consideration of
the use of a bespoke SMS system in a psychiatric hostel. Reflections on the study not only
highlight the ways in which the system entered and facilitated collaboration in the hostel, but
also demonstrate the utility of developing tools supporting collaboration in ubiquitous
computing environments into tools that support social science research as well. These ‘tools’
– i.e., the records generated through the use of the system - do not stand alone or in isolation
but rely for their utility and meaning on their marriage to other ethnographic means of
inquiry, which highlights the mutually supportive nature of novel and established methods
that make the social character of action and technology use visible.
It is not that digital records are inherently mutually supportive, however – they have to be
made mutually supportive, they have to be ‘worked upon’ to transform them into resources
that support the ethnographic enterprise. We have sought to unpack the work that takes to
make digital records work by explicating the kinds of operations that ethnographers perform
on them. This, in turn, highlighted a number of key requirements for support, including
marrying resources generated within or internal to ubiquitous computing environments (e.g.,
text records and audio files) to resources external to that environment, such as video;
selecting relevant parts from digital records that accord with external resources and
temporally synchronizing; stripping non-relevant features out of digital records; adding
content from other resources internal and external, such as audio or video transcripts, to the
record; and reordering records to reflect the interactional order of events rather than the
recorded order of events.
These requirements have informed the development of a dedicated Replay Tool, which
enables ethnographers to visualize the data content of digital records, to extract relevant
sequences of interaction and collaboration, to marry internal and external resources together,
to add content through annotation, and to reorder records to reflect the real world, real time
unfolding of collaboration in ubiquitous computing environments. The development of the
Replay Tool has also highlighted a number of future challenges that it is essential to address,
including the development of spatial visualizations for indexing digital records and spatial
mechanisms for recording them in the first place; the development of mechanisms that enable
ethnographers to better handle temporal issues encountered when working with digital
records, such as representing interactional order and managing temporal slippage in
recordings; and the development of mechanisms that enable ethnographers to perform more
structured forms of analysis. While nascent, the use of digital records and the development of
the Replay Tool demonstrate the potential of this form of e-Research to support social science
research in a novel and demanding domain of inquiry and we look forwards to reporting
further advances in the field in future work.
The research on which this article is based was funded by the UK ESRC e-Social Science Research Node DReSS
(www.ncess.ac.uk/digitalrecord); the UK EPSRC Equator Interdisciplinary Research Collaboration Equator
(www.equator.ac.uk); the Department of Information Systems at the University of Melbourne
(http://www.dis.unimelb.edu.au); and the Smart Internet Technology Cooperative Research Centre, Australia
Anderson, R. and Sharrock W. (1993) “Can organisations afford knowledge”, Computer Supported Cooperative
Work: The Journal of Collaborative Computing, vol. 1 (3), pp. 143-161.
Barkhuus, L., Chalmers, M., Tennent, P., Hall, M., Bell, M., Sherwood, S. and Brown, B. (2005) “Picking
pockets on the lawn”, Proceedings of 7th International Conference on Ubiquitous Computing, pp. 358 - 374,
Benford, S., Seagar, W., Flintham, M., Anastasi, R., Rowland, D., Humble, J., Stanton, D., Bowers, J.,
Tandavanitj, N., Adams, M., Row Farr, J., Oldroyd, A. and Sutton, J. (2004) “The error of our ways: the
experience of self-reported position in a location-based game”, Proceedings of 6th International Conference
on Ubiquitous Computing, pp. 70-87, Nottingham: Springer.
Benford, S., Crabtree, A., Reeves, S., Flintham, M. and Drozd, A. (2006) “Designing for the opportunities and
risks of staging digital experiences in public settings”, Proceedings of the 2006 ACM CHI Conference on
Human Factors in Computing Systems, pp. 427-536, Montreal: ACM.
Benson, D. and Hughes, J.A (1991) “Method: evidence and inference”, Ethnomethodology and the Human
Sciences (ed. Button, G.), pp. 109-136, Cambridge: Cambridge University Press.
Brown, B., MacColl, I., Chalmers, M., Galani, A., Randell, C. and Steed, A. (2003) “Lessons from the
lighthouse”, Proceedings of the 2003 CHI Conference on Human Factors in Computing Systems, pp. 577-
585, Florida: ACM Press.
Button, G. (1992) “The curious case of the vanishing technology”, Technology in Working Order: Studies of
Work, Interaction, and Technology (ed. Button, G.), pp. 10-28, London: Routledge.
Cheverst, K., Clarke, K., Fitton, D., Rouncefield, M., Crabtree, A. and Hemmings, T. (2003a) “SPAM on the
menu: the practical use of remote messaging in community care”, Proceedings of the 2003 ACM Conference
on Universal Usability, pp. 23-29, Vancouver, Canada: ACM Press.
Cheverst, K., Dix, A., Fitton, D. and Rouncefield, M. (2003b) “‘Out to lunch’ - exploring the sharing of personal
context through office door displays”, Proceedings of OZCHI 2003, pp. 74-83, Brisbane, Australia:
Ergonomics Society of Australia.
Crabtree, A., O’Brien, J., Nichols, D., Rouncefield, M., and Twidale, M. (2000) “Ethnomethodologically
informed ethnography and information systems design”, Journal of the American Society for Information
Science, vol. 51 (7), pp. 666-682.
Crabtree, A. (2003) Designing Collaborative Systems: A Practical Guide to Ethnography, London: Springer.
Crabtree, A. (2004) “Taking technomethodology seriously: hybrid change in the ethnomethodology-design
relationship”, European Journal of Information Systems, vol. 13, pp. 195-209.
Crabtree, A., Benford, S., Rodden, T., Greenhalgh, C., Flintham, M., Anastasi, R., Drozd, A., Adams, M., Row-
Farr, J., Tandavanitj, N. and Steed, A. (2004) “Orchestrating a mixed reality game ‘on the ground’”.
Proceedings of the 2004 CHI Conference on Human Factors in Computing Systems, pp. 391-398, Vienna:
Crabtree, A., Rodden, T. and Benford, S. (2005) “Moving with the times: IT research and the boundaries of
CSCW”, Computer Supported Cooperative Work: The Journal of Collaborative Computing, vol. 14 (3), pp.
Crabtree, A. and Rouncefield, M. (2005) “Working with text logs: some early experiences of e-social science in
the field”, Proceedings of the 1st International Conference on e-Social Science, June 22-24, Manchester:
Crabtree, A., Benford, S., Greenhalgh, C., French, A., Chalmers, M., Brown, B. and Tennent, P. (2006a)
“Supporting ethnographic studies of computing ‘in the wild’”, to appear Proceedings of the 2006 ACM
Symposium on Designing Interactive Systems, June 26-28, Montreal: ACM.
Crabtree, A., French, A., Greenhalgh, C., Rodden, T. and Benford, S. (2006b) “Working with digital records:
developing tool support”, to appear Proceedings of the 2nd International Conference on e-Social Science,
June 28-30, Manchester: ESRC.
Erickson, T. and Kellogg, W. (2000) “Social translucence – an approach to designing systems that support social
processes”, ACM Transactions in Computer-Human Interaction, vol. 7, pp. 59-83
Flintham, M., Anastasi, R., Benford, S., Hemmings, T., Crabtree, A., Greenhalgh, C., Rodden, T., Tandavanitj,
N., Adams, M. and Row-Farr, J. (2003) “Where on-line meets on-the-streets: experiences with mobile mixed
reality games”, Proceedings of the 2003 CHI Conference on Human Factors in Computing Systems, pp. 569-
576, Florida: ACM Press.
Foster, I., Kesselman, C. and Tuecke, S. (2001). “The anatomy of the Grid: enabling scalable virtual
organizations”, International Journal of High Performance Computing Applications, vol. 15 (3), pp. 200-
Garfinkel, H. (1967) Studies in Ethnomethodology, Englewood Cliffs, New Jersey: Prentice-Hall.
Gaver, W.H., Dunne, A. and Pacenti, E. (1999) “Design: cultural probes”, Interactions, vol. 6 (1), pp. 21-29.
Graham, C., Cheverst, K. and Rouncefield, M. (2005) “Technologies for the humdrum: trajectories, interactional
needs and a care setting”, Proceedings of OZCHI 2005, pp. 21-25, Canberra: Ergonomics Society of
Heath, C. and J. Hindmarsh (2002) “Analyzing interaction: video, ethnography and situated conduct”,
Qualitative Research in Action (ed. May, T.), pp. 99-121, London: Sage.
Hine, C. (1998) Virtual Ethnography, London: Sage.
Hughes, J. and Sharrock, W. (1997) The Philosophy of Social Research (3rd edition), London: Longman.
Hughes, J., Rodden, T., O’Brien. J., Rouncefield, M. and Blythin, S. (1997) “Designing with ethnography: a
presentation framework for design”, Proceedings of the 1997 ACM Symposium on Designing Interactive
Systems, pp. 147-158, Amsterdam: ACM Press.
Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B.B, Druin, A., Plaisant, C., Beaudouin-Lafon, M.,
Conversy, S., Evans, H., Hansen, H., Roussel, N. and Eiderbäck, B. (2003) “Technology probes: inspiring
design for and with families”, Proceedings of the 2003 CHI Conference on Human Factors in Computing
Systems, pp. 17-24, Florida: ACM Press.
Intille, S., Tapia, E.M., Rondoni, J., Beaudin, J., Kukla, C., Agarwal, S., Bao, L. and Larson, K. (2003) “Tools
for studying behavior and technology in natural settings”, Proceedings of the 5th International Conference on
Ubiquitous Computing, pp. 157-174, Seattle: Springer. ?
Ivory, M. and Hearst, M. (2001) “The state of the art in automating usability evaluation”, ACM Computing
Surveys, vol. 33 (4), pp. 470-516.
Kort, J. and de Poot, H. (2005) “Usage analysis: combining logging and qualitative methods”, Proceedings of
the 2005 CHI Conference on Human Factors in Computing Systems (Extended Abstracts), pp. 2121-2122,
Vienna: ACM Press.
Lynch, M. (2000) “Against reflexivity as an academic virtue and source of privileged knowledge”, Theory,
Culture, and Society, vol. 17, pp. 27-53.
Nardi, B., Whittaker, S. and Bradner, E. (2000) “Interaction and outeraction: instant messaging in action”,
Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, pp. 79-88,
Philadelphia: ACM Press.
Park, R.E., Burgess, E.W. and McKenzie, R.D. (1925) The City, Chicago: University of Chicago Press.
Plummer, K (1983) Documents of Life: An Introduction to the Problems and Literature of a Humanistic Method,
London: Allen & Unwin.
Prosser, J. (ed.) (1998) Image Based Research: A Sourcebook for Qualitative Researchers, London: Falmer
Rouncefield, M., Crabtree, A., Hemmings, T., Rodden, T., Cheverst, K., Clarke, K., Dewsbury, G. and Hughes,
J. (2003) “Designing with care: adapting cultural probes to inform design in sensitive settings”, Proceedings
of OZCHI 2003, pp. 4-13, Brisbane, Australia: Ergonomics Society of Australia.
Sacks, H., Schegloff, E., and Jefferson, G. (1974) “A simplest systematics for the organization of turn-taking in
conversation”, Language, vol. 50, pp. 696-735.
Sacks, H. (1992) “A single instance of a phone-call opening”, Lectures on Conversation (ed. Jefferson, G.),
Volume II, pp. 542-553, Oxford: Blackwell.
Schmidt, K. and Bannon, L. (1992) “Taking CSCW seriously: supporting articulation work”, Computer
Supported Cooperative Work: The Journal of Collaborative Computing, vol. 1 (1), pp. 7-40.
Sharrock, W. and Button, G. (1991) “The social actor: social action in real time”, Ethnomethodology and the
Human Sciences (ed. Button, G.), pp. 137-175, Cambridge: Cambridge University Press.
Strauss, A. (1987) Qualitative Analysis for Social Scientists, Cambridge University Press: Cambridge,
Suchman, L. (1987) Plans and Situated Actions: The Problem of Human-Machine Communication, Cambridge:
Cambridge University Press.
Tapia, E. M., Intille, S. and Larson, K. (2004) “Activity recognition in the home setting using simple and
ubiquitous sensors”, Proceedings of the 2nd International Conference on Pervasive Computing, pp. 158-175,
Wieder, D. and Zimmerman, D. (1977) “The diary: diary interview methods”, Urban Life, vol. 5 (4), pp. 479-
49 Download full-text
Woodruff, A., Szymanski, M., Grinter, R. and Aoki, P. (2002) “Practical strategies for integrating a conversation
analyst in an iterative design process”, Proceedings of the 2002 ACM Symposium on Designing Interactive
Systems, pp. 255-264, London: ACM Press.