Research ProposalPDF Available

Machine Vision in Everyday Life: Playful Interactions with Visual Technologies in Digital Art, Games, Narratives and Social Media

Authors:

Abstract

The project summary part of B1 in my successful ERC-CoG application, which was awarded €2 million. The project will run from August 2018-July 2023.
Rettberg Part B1 MACHINE VISION
1
ERC Consolidator Grant 2017
Research proposal [Part B1]
Machine Vision in Everyday Life:
Playful Interactions with Visual Technologies in
Digital Art, Games, Narratives and Social Media
MACHINE VISION
- Principal Investigator (PI): Jill Walker Rettberg
- PI’s host institution: University of Bergen
- Proposal duration in months: 60 months
In the last decade, machine vision has become part of the everyday life of ordinary people. Smartphones
have advanced image manipulation capabilities, social media use image recognition algorithms to sort and
filter visual content, and games, narratives and art increasingly represent and use machine vision techniques
such as facial recognition algorithms, eye-tracking and virtual reality.
The ubiquity of machine vision in ordinary peoples’ lives marks a qualitative shift where once theoretical
questions are now immediately relevant to the lived experience of ordinary people.
MACHINE VISION will develop a theory of how everyday machine vision affects the way ordinary people
understand themselves and their world through 1) analyses of digital art, games and narratives that use
machine vision as theme or interface, and 2) ethnographic studies of users of consumer-grade machine vision
apps in social media and personal communication. Three main research questions address 1) new kinds of
agency and subjectivity; 2) visual data as malleable; 3) values and biases.
MACHINE VISION fills a research gap on the cultural, aesthetic and ethical effects of machine vision.
Current research on machine vision is skewed, with extensive computer science research and rapid
development and adaptation of new technologies. Cultural research primarily focuses on systemic issues
(e.g. surveillance) and professional use (e.g. scientific imaging). Aesthetic theories (e.g. in cinema theory)
are valuable but mostly address 20th century technologies. Analyses of current technologies are fragmented
and lack a cohesive theory or model.
MACHINE VISION challenges existing research and develops new empirical analyses and a cohesive theory
of everyday machine vision. This project is a needed leap in visual aesthetic research. MACHINE VISION
will also impact technical R&D on machine vision, enabling the design of technologies that are ethical, just
and democratic.
Soviet filmmaker Dziga Vertov was an
early theorist of machine vision. Still from
Man With a Movie Camera (1929).
Screenshot from Republique (2013), a
game where the player sees through
surveillance cameras.
Advertisement for Spectacles, video-
recording sunglasses that can publish
directly to Snapchat.
Rettberg Part B1 MACHINE VISION
2
Section a: Extended Synopsis of the scientific proposal
Snapchat’s selfie lenses are one of many examples of everyday machine vision. Snapchat was launched in
2011, and selfie lenses were introduced less than two years ago, but have become popular with extreme
speed: in all probability, more than 10% of all Norwegians play with selfie lenses every single day
1
. Selfie
lenses are fun. They also normalise biometrics, a form of machine vision that is otherwise used to identify
and control individuals (Rettberg
2017). When you launch the
Snapchat app on your phone, it
opens to the camera, showing
you the image of your own face.
If you hold down a finger on the
image of your face, a biometric
grid is superimposed on your
face, referencing a visual
rhetoric more commonly used in
representations of identification,
surveillance and control. Icons at the bottom of the screen show the different lenses you can apply, and by
swiping through them, you can instantly see your face altered in a dozen ways or more. The facial
recognition algorithms underlying Snapchat’s selfie lens technology allow the distortions and masks to
perfectly align with the user’s own eyes, nose and mouth, so you can move with the lens, seeing for a
moment what it would be like to have rabbit ears and whiskers, or divinely smooth skin, large eyes and a
flower crown. Pass a phone to a child or adult and let them play with the selfie lenses: they will almost
always laugh with delight.
MACHINE VISION investigates how ordinary people interact with machine vision (the registration,
analysis and representation of visual information by machines and algorithms) in their everyday lives. While
existing research has tended to emphasise system-level use of machine vision, for instance in scientific
imaging or surveillance, I will instead examine its domestication and the playful nature of individuals’
interaction with machine vision. MACHINE VISION is founded upon the assumption that art and aesthetic
expression deeply engage with changes in perceptual technologies, and that the study of art, narratives and
games will be a productive source for insights about our relationships with machine vision, and so my main
empirical data will be digital art, games and narratives that thematise and simulate machine vision,
using digital methods and network analysis to gain an overview of a corpus, followed by close readings of
a selection of works. This will be complemented by ethnographic fieldwork studying how ordinary people
use machine vision apps and technologies for self-representation and communication in social media, and by
a thorough survey of the history and theories of pre-digital machine vision. My overall objective is to
develop a theory that explains how the algorithmic machine vision of the 21st century affects the way
ordinary people see themselves and understand the world. Further objectives are specified under the
individual research tasks on page 4-5.
Ground-breaking Nature of the Research: Visual technologies have undergone radical changes in the last
decades and are still changing rapidly, creating a pressing need for new theories that build upon and go well
beyond earlier work to understand a world where machine vision is not only ubiquitous, but where the
majority of the population have access to machine vision as creators and publishers, not just consumers.
Although there is a great deal of technological research and development on machine vision and computer
vision in computer science, there is a large research gap on how this cultural shift to seeing with and through
technology affects us as a society and as individuals. My research is unique in its focus on the personal use
of machine vision, and in my three research questions I focus on shifts in agency, on the perceived
objectivity of images, and on the values embedded in machine vision. This is high-risk research, because of
its cross-disciplinary methodology (combining aesthetic analysis with digital methods and ethnographic
research), novel object of study and innovative combination of theory and empirical analysis, but it is also
extremely high-gain. 10 years from now, humanities scholars will use MACHINE VISION’s findings to
analyse digital visual technologies with the precision we now only can apply to older media, and computer
scientists will use its insights to develop technology that supports a democratic, ethical and just society.
1
A survey in late 2016 showed that 36% of Norwegian residents over 18 use Snapchat every day (Ipsos 2016), and
according to Snap Inc., over a third of Snapchat’s daily users play with selfie lenses every day (Snap Inc. 2017).
Figure 1: Selfie lenses on Snapchat, February 2017. Image on left shows biometric grid.
Rettberg Part B1 MACHINE VISION
3
Research Questions: I have three main research questions, addressing agency, objectivity and values:
RQ1) Which kinds of agency develop and which are limited in the interaction between individual users and
machine vision? I hypothesise that posthumanist theories of distributed agency/subjectivity between
humans and machines (Hayles 1999; Braidotti 2013; Flanagan 2014; Nayar 2014) can be applied to
ordinary peoples’ interactions with machine vision, but this has not yet been tested. Playing with selfie
lenses or a digital camera app with filters and photo editing software gives ordinary users a level of control
over the way that they are represented that was previously only available to expert users and artists, and yet
there are many ways in which their creativity and agency are constrained. Games, art works and narratives
provide particularly rich material to understand constraints upon agency, with their richly imagined dystopias
but also through player/viewer/user interfaces that simulate machine vision. Ethnographic research will
provide interpretations of user’s experiences of their interactions with and through machine vision. In
addition to posthumanist theories, I will also consider other approaches to agency, such as Actor-Network
Theory (Latour 2007; Law and Hassard 1999), postphenomenology (Ihde 1990; Rosenberger and Verbeek
2015) and the photography theories of Flusser (2000), Virilio (1994) and others.
RQ2) Does the experience of being able to manipulate the visual as data through everyday interaction with
machine vision lead us to see the world and ourselves as malleable? Many scholars have argued that seeing
the world as data leads us to believe in the objectivity or truth status of data above our own perception. José
van Dijck calls this “almost ideological belief in data” dataism (Dijck 2014). A high-risk element of this
project is that I will explore an alternative to the currently dominant theory of dataism: my hypothesis is
that the malleability and alternative visualisation possibilities of visual representations support a post-
truth society where we do not trust visual or ultimately, perhaps, our own perceptions, which are
increasingly enmeshed with machine vision. The idea of vision as a more reliable sense than others
(“seeing is believing”) is connected to our centuries-long assumption that vision is passive. This relates to
long-standing discussions on the status of representation in the philosophy of science (Hacking 1983) as well
as in influential writings on art and culture (Debord 1967; Baudrillard 1981). What has changed since their
work is not simply the immense increase in visual display and visual communication, but more importantly
that machine vision is part of the visual vernacular today. We all create and manipulate images using
machines, and the experience of altering or choosing between equally valid visual representations (for
instance by considering which filter to apply to an image before sharing it) is becoming common for ordinary
people and not only experts. Machine vision is no longer left to specialists, and this means ordinary people
have an embodied knowledge that makes today’s experience of the malleability of reality qualitatively very
different from postmodern theoretical understandings. If so, this has great impact on our understanding of
current discussions about post-factual and post-truth politics, where images frequently are used to support
claims.
RQ3) Which values are embedded in machine vision, and which biases are introduced or supported? Racial
bias and biases against other groups are often coded into technology, intentionally or unintentionally (Winner
1980). For instance, 20th century analogue film was calibrated to show nuances in light-skinned but not dark-
skinned faces (Roth 2009), and image recognition algorithms have misrecognised African faces as “gorillas”.
My hypothesis is that everyday machine vision privileges certain kinds of subject. This research
question therefore asks how machine vision shapes its users: What types of bodies or behaviours are
supported or constrained by different kinds of personal machine vision? What kind of role does a work of
digital art or a computer game require you to perform? Do different groups of people (e.g. refugees vs
university students, women vs men, young vs old, Asian vs European, Global North vs Global South)
experience personal machine vision in the same way? To answer these questions, the project will draw upon
research in areas such as critical algorithm studies (Gillespie and Seaver 2015), feminist and queer theory,
disability studies and post-colonial theory, as well as applying narratology and ludology to understand the
position of the user in interacting with relevant apps, games, narratives and artworks.
State of the Art
My interest in machine vision began with my work on selfies and self-tracking for my book Seeing
Ourselves Through Technology: How We Use Selfies, Blogs and Wearable Devices to See and Shape
Ourselves (2014b). At first I saw selfies primarily as a parallel to the self-representations and self-narrations
I had studied previously in blogs and electronic literature (2014a; 2017b; 2009; Walker 2005a), but as I
analysed selfie apps and “beautifying” filters, as well as the sharing and liking ecosystems selfies are
embedded in, I realised that the algorithmic manipulation of images that is inherent to digital technology is
deeply entwined with our self-understanding. Building upon this work, I wrote a chapter on “Biometric
Citizens” that further examines the connections between selfies and machine vision (Rettberg 2017a), and
the way these are embedded within large-scale systems of power and surveillance. I also developed my
interest in artistic explorations of the topic through two conference papers discussing art works that
Rettberg Part B1 MACHINE VISION
4
interrogate machine vision (Rettberg 2016a; Rettberg 2016b), building upon my earlier work on digital art
(Walker 2003), games (Corneliussen and Rettberg 2008), and electronic literature (Walker 2004; 2005a).
Digital cameras are one of the key machine vision technologies ordinary users engage with, and the
history and theories of photography and cinema provide valuable perspectives on my three research
questions. Algorithmic images and machine vision are only the latest of a succession of visual technologies
ranging from the mathematical models of Alberti’s perspective to the camera obscura, magic lanterns,
kaleidoscopes, photography, stereoscopes, cinema, video, x-rays, magnetoscopes, virtual reality headsets,
drones, airport security systems, home surveillance systems and satellite mapping. Artists and philosophers
have long argued that the way we see the world, the way we are able to see the world, is deeply linked to
the technologies and techniques we use to see it. “Imagine an eye unruled by man-made laws of
perspective, an eye unprejudiced by compositional logic,” film-maker Stan Brakhage wrote in 1962. We
have learned these ways of seeing since birth. We base our self-understanding on the media technologies of
our time, as Kittler noted when he pointed out that Plato described the human soul as a wax tablet, while we
after the introduction of cinema imagine that just before death, we see our lives play before our eyes like a
film (Kittler 2009, 49). It is likewise well established that major shifts in representational technologies
have profound effects on human culture. The introduction of linear perspective in the 15th century changed
our relationship to religion and made enlightenment technologies possible (Edgerton 2009), and the
introduction of print changed the way we share and create knowledge (Eisenstein 1979).
Histories of visual technology that focus on art, cinema and photography generally tell the story as
leading up to a specific technology, such as perspective (Kemp 1992; Edgerton 2009), biometrics (Gates
2011) or digital color (Kane 2014), thus leaving out points that are key for machine vision in a broader sense,
or, they stop short too early, as is the case with Kittler’s Optical Media, written in 1999 (2009). Other
relevant research includes theoretical work on the visual (Vertov 1984 orig. 1923; Virilio 1994; Flusser
2000; Friedberg 2006), on new media studies and digital culture (Manovich 1993; Kane 2014; Uricchio
2011; Losh 2015), on research on scientific imaging and visualisation within science and technology studies
(STS) and media aesthetics (Coopmans et al. 2014; Carusi and Hoel 2014; Hausken 2007; Hausken 2015) as
well as research on surveillance images and biometrics (Introna and Wood 2004; Gates 2011; Greene 2015).
I will explore contemporary philosophical approaches to machine vision (e.g. Agamben 2011), and will draw
upon research on the interaction between visual technologies and literature (Danius 2002). Research on
algorithmic bias and the inscription of certain expectations for identity in technologies (Roth 2009; Chun
2011; Gray 2014) will also be important. Within internet studies and digital anthropology there is a growing
interested in selfies, frequently discussed in terms of performance and identity (Burns 2015; Tiidenberg
2015; Tiidenberg 2014; Seko and Lewis 2016).
What is lacking is a comprehensive theory that combines the aesthetic insights of visual theorists
with contemporary cultural expression as it thematises and simulates machine vision, and with empirical
findings of how actual users experience everyday machine vision.
Methodology
The question of how everyday machine vision affects us is too complex to be answered through incremental
efforts, and requires a large-scale, ambitious and integrative research framework. The research will be
organised in five research tasks, each of which will provide a different kind of answer to the three main
research questions. Task 1 provides the necessary background for the other tasks. Task 2 and 3 analyse art,
games and narratives, with Task 2 developing the corpus that Task 3 is dependent on. Task 4 runs in parallel
to 3, providing the ethnographic study of ordinary users’ playful interactions with machine vision. Task 5
synthesizes findings from the other tasks and develops a comprehensive theory of everyday machine vision.
Research Task 1: Survey of History and Theories of Machine Vision
Objective: Compare and synthesise existing histories and theories of machine vision, especially within
scholarship on visual art and visual communication, but also considering theories from other disciplines,
such as society and technology studies, surveillance studies, internet studies and computer science.
Short Description: This task will primarily be conducted by me, with input from the PhD candidates, who
will devote a chapter of their dissertations to writing about the history and theories of machine vision within
their field (art, games or narrative). The emphasis will be on examining how existing historical accounts and
theories discuss issues that are key to my research questions: agency (RQ1), objectivity (RQ2) and values
(RQ3).
Research Task 2: Distant Reading of Art, Narratives and Games that Thematise Machine Vision
Objective: Gain an overview of how machine vision is represented in digital art, games and narratives, and
of how it is used as interface. Use network analysis to see whether there are correlations between agency
(RQ1), objectivity (RQ2) and values (RQ3) and particular genres, modes of interaction etc.
Rettberg Part B1 MACHINE VISION
5
Short Description: My team will use keywords to search public databases (e.g. IMDb, Steam, Rhizome) to
identify a representative corpus of games, digital art and narratives from a set time period that thematise or
simulate machine vision. We will enter information about the works in the ELMCIP Knowledge Base, a
relational database developed in the Electronic Literature Research Group at UiB for a HERA-funded project
I was co-investigator on. We will preserve existing metadata and also develop a set of tags to describe
features of the works relevant to the research questions. We will use distant reading techniques (Moretti
2005; 2013) and more specifically, network analysis, to visualise and analyse shared and differing
representations of machine vision in various works, using a range of bi-partite network views clustering
different sets of characteristics of the works to search for patterns. The ELMCIP Knowledge Base is well-
suited for my purpose (S. Rettberg and Rasmussen 2014), with the open source visualisation software Gephi
for the network analysis and visualisation. This is novel application of a basic methodology I have
experience with (Rettberg 2014c; Berry et al. 2015). There is a risk that I will not be able to identify a
representative corpus, as the public databases my team will use as sources have very different levels of
completeness. There is also a risk that the analysis will not find correlations. If I can achieve this, though, it
will be an extremely valuable component of the project and a genuinely new way of gaining a big
picture view of how art, games and narratives are situating machine vision today.
Research Task 3: Close Readings of Selected Art, Games and Narratives
Objective: Provide analyses of how agency (RQ1), objectivity (RQ2) and values (RQ3) come into play in a
selection of digital art, games and narratives that thematise or simulate machine vision.
Short Description: Selected works from the corpus gathered in Research Task 2, will be analysed,
examining the way that they represent machine vision in order to understand how they position the viewer,
player or reader in relation to the machine vision. Each PhD project will look at machine vision as theme and
as interface in the genre studied. For instance, the movie Minority Report (2002) addresses machine vision as
a theme, showing consumers constantly identified and addressed by commercial software advertisements. In
a dramatic scene, the protagonist has his retina removed so as to avoid biometric identification. In games like
WATCH_DOGS (Ubisoft 2014) and Republique (Camoflaj 2013-16) players see and interact with the
gameworld through surveillance cameras, while Pokemon Go (2016) uses augmented reality. Danny
Cannizzaro & Samantha Gorman’s PRY (2014), a visual and narrative work of electronic literature, makes
the reader use their fingers on the touch screen to pry open the protagonist’s eyes to access the story, while
digital art installations and many games use facial recognition algorithms or eye-tracking to integrate the
viewer into the artwork. The analyses will draw upon the respective disciplines and methodologies of 1) art
history and visual studies, 2) electronic literature studies, comparative literature, narratology and cinema
studies and 3) game studies and ludology, with particular emphasis on the three main research questions.
Research Task 4: An Ethnographic Study of Users and Developers of Machine Vision
Objective: Develop an understanding of how individual users of machine vision technologies in media
experience agency (RQ1), the relationship between visual representations and their perception of themselves
and others (RQ2), and values/bias in machine vision (RQ3), and how developers of the technology consider
values and bias when designing everyday machine vision technologies (RQ3).
Short Description: Mobile phone technology will be used as the hub for ethnographic research involving
participant observation of online and offline communities of avid phone photographers, and in-depth
interviews with 15-20 participants, including both users and developers of machine vision technology for
mobile phone photography such as filters, algorithmic image manipulation and facial recognition software.
Ethnographers and photography theorists have studied selfies and self-representation, but have focused on
issues like societal power relations (Abidin 2016; Burns 2015), human-to-human communication (Frosh
2015) and body image (Tiidenberg 2014) and not specifically on taking and sharing selfies as a form of
interaction with and through machine vision technology. An anthropologist or ethnographer with experience
in digital anthropology (likely recruited from the Association of Internet Researchers) will conduct this part
of the research, which will involve participant observation, in-depth interviews, and analysis of visual
material.
Research Task 5: Consolidate and develop integrative theory of machine vision
Objective: Synthesise results from the first four research tasks to provide comprehensive answers to the
three research questions, resulting in an integrative theory of machine vision.
Short Description: Building upon the historical and theoretical work, as well as the empirical analysis and
distant reading, I will identify shared themes or apparently contradictory findings to develop a novel and
comprehensive theory of machine vision’s effect on our understanding of self and our surroundings.
Rettberg Part B1 MACHINE VISION
6
Feasibility
High-Risk Nature of the Research: This is the first attempt to develop a theory of a new technology,
and is thus inherently risky. Beyond this, the cross-disciplinary methodology and the goal of incorporating
prior theory are high-risk. MACHINE VISION is radically cross-disciplinary in that it combines methods
from the digital humanities and network analysis with aesthetic analysis and ethnographic fieldwork.
Combining ethnographic and aesthetic research is uncommon and will require careful management to ensure
it is mutually productive and not simply two separate strands of research. MACHINE VISION also aims to
draw upon theories from computer science, and to develop research that is valuable for these disciplines as
well as for the humanities, requiring networking and an ability to understand and communicate to other
disciplines. The use of network analysis in Research Task 2 is also a risk factor as this method is not yet
well established in cultural analyses, and while I hope it will provide valuable answers to the research
questions and a model for a new methodology, it may also give no more than a useful overview and basic
information about correlation. Further, prior theory (e.g. posthumanism (RQ1) or visual theories (Research
Task 1 and State of the Art)) may not support the empirical analyses. MACHINE VISION will in any case
develop new theory since today’s ubiquitous machine vision is qualitatively different from previous machine
vision, but while I aim to go well beyond earlier work, I do hope to build upon it. If this is not possible, I will
test theories from other domains, and develop new models to explain the empirical findings.
My capability to lead the project: My track record shows that I am consistently the first or among the
first to study digital cultural phenomena, including blogging (Mortensen and Walker 2002), selfies
(Walker 2005b), transmedia storytelling (Walker 2005a), self-tracking (Rettberg 2014b) and Snapchat
(Rettberg 2017c; Rettberg 2017). I am an excellent networker, both in social media (see my blog jilltxt.net
and Twitter and Snapchat accounts @jilltxt) and in person, which has led to field-building and
collaborations (Corneliussen and Rettberg 2008; Senft et al. 2014). My education and experience provide a
unique multi-disciplinary position that makes me the ideal leader for a project of this scope. My MA was
in comparative literature, and my early research was on digital art, electronic literature and games. My first
research publication was at the Association of Computing Machinery (ACM) Hypertext conference, where
my literary analysis of a hypertext fiction won the award for Best Paper by a Newcomer (Walker 1999), and
where I found the mixture of computer scientists and literary critics extremely productive. Now I regularly
attend the Association of Internet Researchers conferences, and have collaborated with scholars from this
community (Albury et al. 2017; Warfield et al. 2016), giving me a knowledge of ethnographic approaches to
understanding digital culture.
Research Environment: I lead UiB’s Digital Culture Research Group, which is a natural home for the
project. I will also work with UiB’s Electronic Literature Research Group, which hosts the ELMCIP KB that
is central to Research Task 2. The PhD student working on games will also participate in the Computer
Games Research Network at UiB, where I am a member. My team will have full administrative support from
the University of Bergen, including research training for PhD candidates and high-quality technical,
informational and organisational infrastructure for all researchers. A Reference Board of top international
scholars with specific expertise on aspects of the project will be established, with participants invited to
annual workshops. I have lived in Bergen for many years, and will be there at least 80% of the time.
High Gain and Multi-Disciplinary Research Impact: MACHINE VISION is a potent response to exactly
the kind of research challenge the humanities must rise to today: We are seeing rapid technological
development and societal adoption of machine vision, with massive research efforts on the topic from
computer science, and the humanities have a wealth of relevant knowledge to contribute. However,
humanities research on machine vision has so far been fragmented and not sufficiently responsive to the
radical technological and cultural developments of the last decades. MACHINE VISION will develop new
knowledge that will give researchers the necessary new tools for interpreting the role of everyday playful
interaction with machine vision in social media, games, narratives and art, thus directly affecting research in
aesthetic disciplines like art history, literary studies, game studies and cinema studies, but also social science
research in media studies, internet studies, anthropology, and on surveillance, human-computer
communication and digital culture in general. It will also inform the technical research and development on
machine and computer vision which is now largely founded in computer science and engineering disciplines.
It will have a societal impact by providing the general public with an understanding of machine vision's
influence on ourselves and on society that enables us to make choices about the technologies we introduce to
our culture, rather than simply embracing that which is technologically possible.
Rettberg Part B1 MACHINE VISION
7
Bibliography
Abidin, Crystal. 2016. “‘Aren’t These Just Young, Rich Women Doing Vain Things Online?’: Influencer
Selfies as Subversive Frivolity.” Social Media + Society 2 (2): 2056305116641342.
doi:10.1177/2056305116641342.
Agamben, Giorgio. 2011. “Identity Without the Person.” In Nudities, 4654. Stanford: Stanford UP.
Albury, Kath, Tama Leaver, Alice Marwick, Jill Walker Rettberg, and Theresa M. Senft. 2017. “The Selfie
Course: More than a MOOC.” In Massive Open Online Courses and Higher Education: Where to
Next?, edited by Rebecca Bennett and Mike Kent. Farnham: Ashgate.
Baudrillard, Jean. 1981. Simulacres et Simulation. Paris: Éditions Galilée.
Berry, David M., Erik Borra, Anne Helmond, Jean-Christophe Plantin, and Jill Walker Rettberg. 2015. “The
Data Sprint Approach: Exploring the Field of Digital Humanities through Amazon’s Application
Programming Interface” 9 (4). http://www.digitalhumanities.org/dhq/vol/9/3/000222/000222.html.
Braidotti, Rosi. 2013. The Posthuman. Cambridge: Polity.
Burns, Anne. 2015. “Self(ie)-Discipline: Social Regulation as Enacted Through the Discussion of
Photographic Practice.” International Journal of Communication 9 (May): 17161733.
Carusi, Annamaria, and Aud Sissel Hoel. 2014. “Toward a New Ontology of Scientific Vision.” In
Representation in Scientific Practice, edited by Catelijne Coopmans, Janet Vertesi, Michael E.
Lynch, and Steve Woolgar, 20121. Cambridge MA: MIT Press.
Chun, Wendy Hui Kyong. 2011. Programmed Visions: Software and Memory. Cambridge, MA, USA: MIT
Press. http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10496266.
Coopmans, Catelijne, Janet Vertesi, Michael E. Lynch, and Steve Woolgar, eds. 2014. Representation in
Scientific Practice Revisited. Cambridge MA: MIT Press.
Corneliussen, Hilde, and Jill Walker Rettberg. 2008. Digital Culture, Play, and Identity: A World of
Warcraft Reader. MIT Press.
Debord, Guy. 1967. La Société Du Spectacle. Paris: Buchet-Chastel.
Dijck, Jose van. 2014. “Datafication, Dataism and Dataveillance: Big Data between Scientific Paradigm and
Ideology.” Surveillance & Society 12 (2): 197208.
Edgerton, Samuel Y. 2009. The Mirror, the Window, and the Telescope: How Renaissance Linear
Perspective Changed Our Vision of the Universe. Ithaca and London: Cornell UP.
Eisenstein, Elizabeth. 1979. The Printing Press as an Agent of Change: Communications and Cultural
Transformations in Early-Modern Europe. Vol. 1. 2 vols. Cambridge: Cambridge UP.
Flanagan, Victoria. 2014. Technology and Identity in Young Adult Fiction: The Posthuman Subject.
Basingbroke: Palgrave Macmillan.
Flusser, Vilém. 2000. Towards a Philosophy of Photography. London: Reaktion Books.
Friedberg, Anne. 2006. The Virtual Window: From Alberti to Microsoft. Cambridge MA: MIT Press.
Frosh, Paul. 2015. “The Gestural Image: The Selfie, Photography Theory, and Kinesthetic Sociability.”
International Journal of Communication 9 (0): 22.
Gates, Kelly. 2011. Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance.
New York: NYU Press.
Gillespie, Tarleton, and Nick Seaver. 2015. “Critical Algorithm Studies: A Reading List.” Social Media
Collective. November 5. https://socialmediacollective.org/reading-lists/critical-algorithm-studies/.
Gray, Kishonna L. 2014. Race, Gender, and Deviance in Xbox Live: Theoretical Perspectives from the
Virtual Margins. New York: Routledge.
Greene, Daniel. 2015. “Drone Vision.” Surveillance & Society 13 (2): 23349.
Hacking, Ian. 1983. Representing and Intervening: Introductory Topics in the Philosophy of Natural Science.
Cambridge University Press.
Hausken, Liv. 2007. “The Aesthetics of X-Ray Imaging.” In Aesthetics at Work, edited by Arne Melberg.
Oslo: Unipub.
———. 2015. “The Media Aesthetics of Brain Imaging in Popular Science.” Leonardo 48 (1).
Hayles, N. Katherine. 1999. How We Became Posthuman. Chicago: University of Chicago Press.
Ihde, Don. 1990. Technology and the Lifeworld: From Garden to Earth. Bloomington: Indiana University
Press.
Introna, Lucas, and David Wood. 2004. “Picturing Algorithmic Surveillance: The Politics of Facial
Recognition Systems.” Surveillance & Society 2 (2/3).
http://library.queensu.ca/ojs/index.php/surveillance-and-society/article/view/3373.
Ipsos. 2016. “Ipsos’ Tracker Om Sosiale Medier Q3’16.” http://ipsos-mmi.no/some-tracker.
Kane, Carolyn L. 2014. Chromatic Algorithms: Synthetic Color, Computer Art, and Aesthetics After Code.
Univ. of Chicago Press.
Rettberg Part B1 MACHINE VISION
8
Kemp, Martin. 1992. Science of Art: Optical Themes in Western Art from Brunelleschi to Seurat. New
Haven: Yale UP.
Kittler, Friedrich. 2009. Optical Media. Cambridge, UK; Malden, MA: Polity.
Latour, Bruno. 2007. Reassembling the Social: An Introduction to Actor-Network-Theory. 1st edition.
Oxford; New York: Oxford University Press.
Law, John, and John Hassard. 1999. Actor Network Theory and After. 1 edition. Oxford England ; Malden,
MA: Wiley-Blackwell.
Losh, Elizabeth. 2015. “Feminism Reads Big Data: ‘Social Physics,’ Atomism, and Selfiecity.” International
Journal of Communication, no. 9: 164759.
Manovich, Lev. 1993. “The Mapping of Space: Perspective, Radar, and 3-D Computer Graphics.”
Manovich.net. http://manovich.net/index.php/projects/article-1993.
Moretti, Franco. 2005. Graphs, Maps, Trees: Abstract Models for a Literary History. London; New York:
Verso.
———. 2013. Distant Reading. Verso Books.
Mortensen, Torill, and Jill Walker. 2002. “Blogging Thoughts: Personal Publication as an Online Research
Tool.” In Researching ICTs in Context, edited by Andrew Morrison, 24979. Oslo: InterMedia,
University of Oslo. http://www.intermedia.uio.no/konferanser/skikt-
02/docs/Researching_ICTs_in_context-Ch11-Mortensen-Walker.pdf.
Nayar, Pramod. 2014. Posthumanism. Cambridge: Polity Press.
Rettberg, Jill Walker. 2009. “‘Freshly Generated for You, and Barack Obama’ How Social Media Represent
Your Life.” European Journal of Communication 24 (4): 45166. doi:10.1177/0267323109345715.
———. 2014a. Blogging. 2nd ed. Cambridge: Polity Press.
———. 2014b. Seeing Ourselves Through Technology: How We Use Selfies, Blogs and Wearable Devices to
See and Shape Ourselves. Basingbroke: Palgrave. Open Access.
———. 2014c. “Visualising Networks of Electronic Literature: Dissertations and the Creative Works They
Cite.” Electronic Book Review, June.
http://www.electronicbookreview.com/thread/electropoetics/analyzing.
———. 2016a. “Machine Vision as Viewed Through Art: Hostile Other or Part of Ourselves?” In Post-
Screen: Intermittence + Interference. Edições Universititárias Lusófonas.
http://jilltxt.net/txt/artmachinevisionPSF2016.pdf.
———. 2016b. “‘Now Objects Perceive Me’: Art That Interrogates Machine Vision.” In Association of
Internet Researchers (AoIR2016). http://bora.uib.no/handle/1956/12897.
———. 2017a. “Biometric Citizens: Adapting Our Selfies To Machine Vision.” In Selfie Citizenship, edited
by Adi Kuntsman. Basingbroke: Palgrave.
———. 2017b. “Self-Representation in Social Media.” In SAGE Handbook of Social Media, edited by Jean
Burgess, Alice Marwick, and Thomas Poell. London: SAGE Publications.
———. 2017c. “Snapchat.” In Appified, edited by Jeremy Wade Morris and Sarah Murray. Ann Arbor:
University of Michigan Press.
———. 2017. How Snapchat Uses Your Face (A Snapchat Research Story). Accessed February 1.
https://www.youtube.com/watch?v=2XOd-rc7r98.
Rettberg, Scott, and Eric Rasmussen. 2014. “The ELMCIP Electronic Literature Knowledge Base: Project
Report.” In Electronic Literature as a Model of Creativity and Innovation in Practice: A Report from
the HERA Joint Research Project, 30753. West Virginia UP.
Rosenberger, Robert, and Peter-Paul Verbeek, eds. 2015. Postphenomenological Investigations: Essays on
HumanTechnology Relations. Lanham: Lexington Books.
Roth, Lorna. 2009. “Looking at Shirley, the Ultimate Norm: Colour Balance, Image Technologies, and
Cognitive Equity.” Canadian Journal of Communication 34 (1). http://www.cjc-
online.ca/index.php/journal/article/view/2196.
Seko, Yukari, and Stephen P. Lewis. 2016. “The Selfharmed, Visualized, and Reblogged: Remaking of
Self-Injury Narratives on Tumblr.” New Media & Society, July, 1461444816660783.
doi:10.1177/1461444816660783.
Senft, Theresa M., Jill Walker Rettberg, Elizabeth Losh, Kath Albury, Radhika Gajjala, Gaby David, Alice
Marwick, et al. 2014. “The Selfie Course.” August 28. http://www.selfieresearchers.com/the-selfie-
course/.
Snap Inc. 2017. “Form S-1 Registration Statement (for Initial Public Offering Filing).” United States
Securities and Exchange Commission.
https://www.sec.gov/Archives/edgar/data/1564408/000119312517029199/d270216ds1.htm#rom270
216_9.
Rettberg Part B1 MACHINE VISION
9
Tiidenberg, Katrin. 2014. “Bringing Sexy Back: Reclaiming the Body Aesthetic via Self-Shooting.”
Cyberpsychology: Journal of Psychosocial Research on Cyberspace 8 (1). doi:10.5817/CP2014-1-3.
———. 2015. “Boundaries and Conflict in a NSFW Community on Tumblr: The Meanings and Uses of
Selfies.” New Media & Society, January, 1461444814567984. doi:10.1177/1461444814567984.
Uricchio, William. 2011. “The Algorithmic Turn: Photosynth, Augmented Reality and the Changing
Implications of the Image.” Visual Studies 26 (1): 2535. doi:10.1080/1472586X.2011.548486.
Vertov, Dziga. 1984. Kino-Eye: The Writings of Dziga Vertov. University of California Press.
Virilio, Paul. 1994. The Vision Machine. Translated by Julie Rose. University of Indiana Press.
Walker, Jill. 1999. “Piecing Together and Tearing Apart: Finding the Story in ‘Afternoon’.” In , edited by
Klaus Tochtermann, Jörg Westbomke, Uffe K. Wiil, and John J. Leggett, 11117. ACM Press.
http://jilltxt.net/txt/afternoon.html.
———. 2003. “Fiction and Interaction: How Clicking a Mouse Can Make You Part of a Fictional World.”
Dr. art. thesis, University of Bergen. http://www.ub.uib.no/elpub/2003/d/517001.
———. 2004. “How I Was Played by Online Caroline.” In First Person: New Media as Story, Performance,
and Game, edited by Noah Fruin and Pat Harrigan. MIT Press.
———. 2005a. “Distributed Narrative: Telling Stories Across Networks.” In The 2005 Association of
Internet Researchers Annual, edited by Mia Consalvo, Jeremy Hunsinger, and Nancy Baym. New
York: Peter Lang.
———. 2005b. “Mirrors and Shadows: The Digital Aestheticisation of Oneself.” In , edited by Lisbeth
Klastrup and Susana Tosca, 18490. IT University.
Warfield, Katie, Crystal Abidin, Natalie Hendry, Jill Walker Rettberg, and Katrin Tiidenberg. 2016.
“Representation, Presentation, Embodiment/Emplacement.” In . Berlin.
Winner, Langdon. 1980. “Do Artifacts Have Politics?” Daedalus 109 (1): 12136.
Conference Paper
Full-text available
Machine vision – the registration, analysis, and representation of visual information by machines and algorithms (Rettberg 2017) – is currently hiding behind videogames’ playful exterior. However, machine vision technologies such as night vision overlays, facial recognition systems, and surveillance cameras have been represented within virtual environments for decades. To bring into the light and acknowledge this technology as an important agent, I build on theorizations of videogames as assemblages of multiple agents (Taylor 2009) and of posthuman interrelated agency (Hayles 2017; Braidotti 2013). This study thus provides an overview of diegetic representations of machine vision in videogames in order to begin an analysis of distributed agency between human and nonhuman agents.
Article
Full-text available
Taking seriously the global trend of selfies becoming marketable and entangled in ecologies of commerce, this article looks at Influencers who have emerged as (semi-)professional selfie-producers and for whom taking selfies is a purposively commercial, thoughtful, and subversive endeavor. Based on in-depth ethnographic fieldwork and grounded theory analysis, I examine Influencers’ engagements with selfies on Instagram and their appropriations of selfies as salable objects, as tacit labor, and as an expression of contrived authenticity and reflexivity. Through these practices, Influencers achieve “subversive frivolity,” which I define as the under-visibilized and under-estimated generative power of an object or practice arising from its (populist) discursive framing as marginal, inconsequential, and unproductive.
Article
Images featuring self-injury (SI) have been proliferating on social media. This article reports the findings of a visual narrative analysis of 294 photo-based posts on Tumblr, exploring how SI is narrated through the interplay between image content, photographic composition, associated texts and tags, and reblogging. Findings reveal a shift in the iconography of SI from direct depictions of self-injured bodies to re-appropriations of popular media content that figuratively represent emotional struggles. Images of self-inflicted wounds received 10 times less reblogs than images without wounds, and media memes conveying hopeless moods were the most widely distributed. These memes represent SI as a form of life struggle virtually anyone can face while complicating conventional readings of SI as an individual pathologic experience. We discuss these findings in the context of an emergent online curation culture and how Tumblr’s unique affordances may both offer and limit possibilities for narrating SI.
Book
"Virilio offers a cool, precise look at an impending future in which reality shall simply cease to exist. Highly recommended." —Choice Surveying art history as well as the technologies of war and urban planning, one of France’s leading intellectuals provides an introduction to a new "logistics of the image."
Chapter
This introductory chapter sets out the theme of representation in scientific practice, and addresses why it remains such a vibrant topic for research in and beyond science and technology studies. The chapter draws particular attention to shifts in the content, context and concepts of inquiry that have occurred since the first volume to bear the name Representation in Scientific Practice, which was edited by Lynch and Woolgar and published by MIT Press in 1990. Like the earlier volume, this new collection of studies includes ethnographic, historical and discourse-analytic approaches to scientific knowledge-making. It revisits representation in scientific practice not only by focusing on novel technologies, sciences, and institutional arrangements that have emerged in recent decades, but also by respecifying recurrent themes in light of novel conceptual and institutional loci for interrogating those themes.
Article
The first of three essays setting out to demonstrate the power of abstract models to revolutionize our understanding of literary history. What do the quantitative curves of novel production tell us about the interplay of markets, politics, sexes, generations, in the life and death of literary forms?.
Book
Race, Gender, and Deviance in Xbox Live provides a much-needed theoretical framework for examining deviant behavior and deviant bodies within one of the largest virtual gaming communities—Xbox Live. Previous research on video games has focused mostly on violence and examining violent behavior resulting from consuming this medium. This limited scope has skewed criminologists’ understanding of video games and video game culture. Xbox Live has proven to be more than just a gaming platform for users. It has evolved into a multimedia entertainment outlet for more than 20 million users. This book examines the nature of social interactions within Xbox Live, which are often riddled with deviant behavior, including but not limited to racism and sexism. The text situates video games within a hegemonic framework deploying whiteness and masculinity as the norm. The experiences of the marginalized bodies are situated within the framework of deviance as they fail to conform to the hegemonic norm and become victims of racism, sexism, and other types of harassment.
Article
For a few weeks in the summer of 2014, a glimmering LED display on the media façade of the FIESP building in São Paulo, Brazil, cycled through hundreds of selfies, thus presenting a multiplicity of images of solo acts of urban self-fashioning now captured by new practices of closely distant mobile photography. A selfie of a young woman with plucked eyebrows and auburn hair transformed into a white man with a receding hairline; his image was supplanted by one of a girl with Asian features; the almond-eyed girl was then replaced by one with long blond tresses. Sunglasses and eyeglasses appeared and disappeared. Facial hair blossomed and receded. Each frame of the SelfieSaoPaulo animation showed an individual citizen's selfie, which was tagged with an approximate age and identified with percentages calibrating gender and degree of smiling affect. The image sequences were aligned with common eye positions, as if to imply that extremely divergent specimens of humanity still shared a common gaze.
Article
"A groundbreaking study. Our Biometric Future considers facial recognition technology through its wide range of political entanglements, such as post-9/11 security measures, the management of urban populations in commercial districts, and self-representation in online social networking sites. Across these contexts, Gates shows how facial recognition's political effects have developed in spite of the fact that the technology does not actually work very well. Written with style and wit, Our Biometric Future will resonate with readers in cultural studies, new media, science and technology studies, and anyone interested in surveillance, privacy and security in contemporary life."