Multimodal genres and transmedia traversals: Social semiotics and the political economy of the sign

Article (PDF Available)inSemiotica 2009(173):283-297 · February 2009with 1,257 Reads
DOI: 10.1515/SEMI.2009.012
Abstract
Multimodal media make meaning by intersecting the semiotic resources of language, visual display, sound and music, cinematic movement, material artifacts, and abstract animation. As meaning-makers, we live across institutions and media and we make meanings that no single medium or institution can control. Transmedia franchises pursue us, colonizing the chronotopes of postmodern life to array the branded content of their media products everywhere we look and go. Social semiotics provides a foundation for critical multimedia analysis that can reach beyond the internal multi-modality of individual works to grasp the social meanings of transmedia franchises and trans-institutional lives. In doing so, I argue that it must re-engage with the larger intellectual project of formulating a political economy of the sign.
Multimodal genres and transmedia traversals:
Social semiotics and the political economy
of the sign
JAY LEMKE
Abstract
Multimodal media make meaning by intersecting the semiotic resources of
language, visual display, sound and music, cinematic movement, material
artifacts, and abstract animation. As meaning-makers, we live across insti-
tutions and media and we make meanings that no single medium or institu-
tion can control. Transmedia franchises pursue us, colonizing the chrono-
topes of postmodern life to array the branded content of their media
products everywhere we look and go. Social semiotics provides a foundation
for critical multimedia analysis that can reach beyond the internal multi-
modality of individual works to grasp the social meanings of transmedia
franchises and trans-institutional live s. In doing so, I argue that it must re-
engage with the larger intellectual project of formulating a political econ-
omy of the sign.
Keywords: multimedia; genre; transmedia; economics; semiotics.
1. Social semiotics
The perspective of social semiotics developed for many of us (e.g., Hodge
and Kress 1988; Lemke 1987; Thibault 1991) out of the social linguistics
of Michael Halliday (1978). Halliday’s approach to language regards it as
a systematic resource for making meaning, a semiotic resource system.
Meanings are made as part of human social activities which define their
context of situation. Which meanings are made, when, how, and by
whom, depend in turn on a wider context of culture. These basic ideas
were developed by Halliday out of the work of his teach er J. R. Firth
(1968), and have their origins in the analyses of cultu ral meaning by the
pioneering ethnographer, Bronislav Malinowski (1923, 1935).
Semiotica 173–1/4 (2009), 283297 00371998/09/0173–0283
DOI 10.1515/SEMI.2009.012 6 Walter de Gruyter
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 283)
The context of situation in which any linguistic meaning is made is
described by the nature of the activity (including the topic of talk), the
social-interpersonal relationships being constructed (with present or im-
agined interlocutors and audiences), and the channels of communication
and their a¤ordances and limitations (print, telephone, speech, email,
etc.). Each of these three major factors tends to be associated with di¤er-
ent sorts of choices from the system s of alternative options for ways of
making meaning that are o¤ered to us by our language. The association
takes the form of a statistical shift in the probability of some options
versus others. Because there is cultural regularity in the combinations of
activities, social relationships, and media in society, so there exist corre-
spondingly identifiable linguistic registers. If an activity plays out in a
way that produces a text (including a written one), then the organiza-
tional regularity of the activity as a sequence of functionally related ac-
tions corresponds to the sequential semantic structure of the text (at least
in the large), which we call its genre structure. The observation that genre
structures result from the meaningful functional organization of action
indicates that action itself is a semiotic resource system, and with it move-
ment, gesture, musical performance, dance, etc. Texts are not the only
traces or relatively enduring material products of action: so also are
drawings, sculptures, artifacts, architectural constructions, etc. The un-
derlying semiotic structure of meaningful human action is the common
denominator that allows all semiotic resource systems to be meaningfully
combined in multimodal genres. Such at least is the view of Hallidayan
social semiotics (e.g., Baldry and Thibault 2005; Kress and Van Leeuwen
2001; Lemke 2005b; O’Halloran 2004; Thibault 2000; Van Leeuwen
1999).
Social semiotics also develops this core model in a particul ar, more spe-
cific way that is quite useful for the practical analysis of multimodal
‘texts.’ For the case of language, Halliday identified three kinds of
functional meaning that every linguistic meaning-making act constructs
and, by generalization, that every act of meaning constructs regardless
of what cultural semiotic system s of interpretation its forms are con-
strued in relation to (e.g., linguistically meaningful forms, gesturally
meaningful forms, graphically meaningful, musically meaningful, etc.).
These generalized semiotic functions (Halliday termed them metafunc-
tions in relation to other, more specific linguistic functions) are, in my
own terminology:
The Presentational function: presenting (or some would say represent-
ing) a state of a¤airs, or enacting a meaningful, usually recognizable
activity (Halliday’s ideational-experiential metafunction)
284 J. Lemke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 284)
The Orientational function: addressing some real or imagined Other,
thereby constructing some kind of social-interpersonal relationship,
with an attitude both toward the Other and toward the presenta-
tional content of one’s own semiotic action (Halliday’s interpersonal-
attitudinal metafunction)
The Organizational function: constructing relationships of parts to
wholes and strands of continuity based on similarity within di¤erence
that tie the whole of the action or activity together and indicate which
elements within it are more closely related to which others, and how
(Halliday’s textual metafunction)
By examining how these three functions are realized in and through each
semiotic modality (language, gesture, music, drawing, etc.) within a
whole multimedia work, and also how they are integrated, not just across
the modalities, but especially across the three general functions, we can
produce very detailed and comprehensive semiotic analyses of multi-
modal phenomena. In doing so, of course, we must appeal to more spe-
cific and ‘delicate’ kinds of meaning distinctions that subserve these very
general functions, and some of these will, naturally, be specific to one
or another of the semiotic modalities. It is only these three most general
semiotic functions that are claimed to apply to all semiotic modalities,
else why would we need so many di¤erent semiotic resource systems?
But the Big Three are enough to ground the integration and interdepen-
dence within multimodal genres of signs produced with man y, very di¤er-
ent semiotic resources.
Before describing this approach in more detail, however, we need to
consider further the common multimodal genres of our culture, in semi-
otic terms. In social semiotics this also means thinking about their history
and their functional role in our social communities. It is another basic
principle of social semiotics that systems of forms have evolved histori-
cally to serve social functions. Even if imperfectly, function is taken to
be a good guide to the identification of relevant features of form, and
the diversity of functions helps us gauge the diversity of meanings which
motivate the di¤er ences among forms (even if they more often det ermine
merely that there shall be di¤erences of form, rather than determining just
what those forms and di ¤erences are). In Bateson’s well-known phrase,
informative meaning is conveyed by ‘di¤erences that make a di¤er ence.’
Since Saussure, we have known that these di¤erences form organized sys-
tems, within which the place of each form determines its potential valeur.
But what happens when signs from di¤erent systems are used to make
meaning together?
Multimodal genres 285
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 285)
2. Multimodal genres: The basics
Printed text is always already multimodal in the sense that its signs are
not simply linguistic but also typographical. No gen re, indeed no sign,
can be strict ly unimodal, except in abstraction from its material realiza-
tion in some material medium of signifiers. Interpreters are free to make
meaning using either, and usually both, the linguistic semiotic relations of
a sign and, in this case, its visual-typographic relations, interpreting it in
both respects as both what it is and what it might hav e been but is not.
More conventionally we think of multimodal genres as those in which
some signifiers are typically interpreted with respect to one semiotic sys-
tem (e.g., language), and other signifiers that are joined to them syntag-
matically are interpreted with respect to another semiotic (e.g., pictorial
display). This raises immediately the basic question of how signs from
two di¤erent semiotic systems can belong to the same syntagm? For ex-
ample, how do we know that a picture on a page is to be interpreted in
relation to some (rather than other) text on that page (or on some other
page)? Multimodal genres grow out of multimodal syntagms (e.g., figure-
and-caption; icon -and-label), and in many cases the latter are possible
precisely because some of the signifiers are interpretable in both semi otics
(e.g., the label on an icon is both a visual sign that can be joined syntag-
matically to the pictorial icon within a graphical syntagm and it is a lin-
guistic sign that can be read as the name or function of that icon).
Visual signs readily form syntagms, and ultimately multimodal genres,
with written linguistic signs, because the latter are also visual signs. While
this is true for visual-textual genres, there is a di¤erent mechanism appar-
ent in, say, the sound-film. No acoustic sign here, whether speech sound
or music sound, is also visual, nor vice versa. How then do film images
form multimodal cinematic syntagms with music or speech? This question
was, of course, first answered by Eisenstein (1943) in great detail, in terms
of two basic mechanisms: temporal synchronicity and cross-modal ho-
mology. The former is more easily understood. By synchronizing speech
with images of moving lips, a joint sign is produced at each instant of
time, consisting of both sound and image, and there is a congruence of
these (which is iconic in some interesting sense, th ough Peirce might
more correctly term it indexical), the same congruence (or redundancy)
which makes lip-reading possible. Some visible lip conformations go with
the production of some sounds. (Is true iconism possible across percep-
tual modalities, or as in this case only a purely indexical-causal relation-
ship?). In the case of music (unless the musical instruments are visi ble, as
with some ‘diegetic’ music in film), normally there is only a certain syn-
chronization of changes in musical (e.g., harmonic-melodic) line with vi-
286 J. Lemke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 286)
sual changes, and Eisenstein discovered the principle of homology, by
which to somehow make these changes also congruent (e.g., a rising me-
lodic line with a rising camera angle, or a pan along a rising series of cli¤s
or mountains). In this case it is not necessarily the presentation of an in-
stantaneous joint-sign (as for speech-sound/lip-shape), but rather the
over-time co-patterning in the two modalities (changes in sound /
changes in image) that provides the multimodal signifier that we can in-
terpret as a unified syntagm.
These examples are, I think, good reminders that syntagms are not
given. They are construed by interpreters. The case of syntagms by ho-
mology of dynamical patterning in time also indicates that signifiers may
have relevant, even essential, extension in time, so that while still mate-
rial, they are material in the sense that processes are material, rather
than in the more limited sense in whic h we imagine things to be material.
Hallidayan social semiotics, following Hjelmslev (1961; see also Lemke
2000a), has developed in this respect a more materialist approach to the
sign than is customary in some other traditions.
Eisenstein also points us towards the role of montage or juxtaposition
e¤ects in the creation of multimedia syntagms. They rely on what psy-
chologists call perceptual closure, but which I think semioticians might
better regard as the irrepressibility of meaning-making. We do not merely
complete basic perceptual patterns; we will make meaningful wholes,
however slight the a¤ordance of the material basis for doing so. Even if
there is no synchronization, no homology or congruence, if signs of the
same or di¤erent semiotics are presented to us in any degree of juxtaposi-
tion in space or time, we eagerly try to construe their meanings as co-
dependent, i.e., we tend to construe syntagms, even multimodal syn-
tagms, at the slightest suggestion. Of course, we do so more fervently as
our cultural experience suggests a greater likelihood that certain kinds of
signs in certain modes of juxtaposition ought to be meaningful. We live in
communities where many kinds of multimodal syntagms are typical and
familiar.
We do, however, still need to learn to make sense of radically new mul-
timodal combinations, as was the case historically for sound-film, or for
artists’ experiments in synesthesia. But we only need to learn to do so in
ways that are consensually recognized in the community. Idiosyncrati-
cally, we are all able to attribute some meaning to almost any juxtaposi-
tion. Hence the Herculean labors of science to persuade us not to make
the many associational meanings (e.g., correspondences, a‰nities, signa-
tures) connecting plants, minerals, heavenly bodies, etc., of centuries past,
as described so well by Foucault at the beginning of Les mots et les choses
(1966).
Multimodal genres 287
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 287)
Conventionalized multimodal genres abound. In our age of inexpensive
image printing, we find images and text arrayed alongside one another,
sometimes in local syntagms and sometimes not (or only connected in
the global syntagm of a page layout or of the genre of a magazine as a
whole). In earlier times, text and image were found on separately printed
pages, and their connections were established by the semantic relations of
a caption to the main text, or by mere juxtaposition on facing pages (but
not overleaf ), or most often by some naming in the text of a recognizable
element of the image. Still earlier of course, the pictorial and decorative
Initials of chapters in holographic manuscripts again made joint signs of
the same inked regions of the page, as both image and letter.
But in between, beginning in the seventeenth century, with the rise of
scientific printed publications, a stricter connection of text and image be-
came necessary. Galileo put his first printed drawing of the rings of
Saturn right in the midst of the words of the page, as a rebus for the miss-
ing word ‘Saturn.’ Very dramatic, this showman! And the image was very
small, just as it would have been seen through his telescope (very iconic,
too!). But more generally, early scientific writers were at pains to link ver-
bal claims to specific numerical (or for botanists, pictorial) evidence. The
terms of the algebraic formulas, integrated into the text in accordance
with their close kinship to language and linguistic argumentation, still
had to be unambiguously linked to the tables of data and results of calcu-
lations made using the formulas, which were not presented as text, but in
tabular form, or later as so-called Cartesian graphs for an example
from Bernoulli, 1734 (see Lemke 1998b). Many new conventions were
created to insure such tight linkages. And so were born the multimodal
genres of natural science and technology.
Scientific research reports today contain not only running text, but typ-
ically one or more graphs, charts, tables, or other specialized visual dis-
plays per page. In brief specialized technical research reports, mainly in
the biosciences, in the prestigious journal Science, averaging 2.5 pages in
length, ther e were an average of no less than six non-textual visual dis-
plays per article, normally accompanied by extensive captions and refer-
ences to the figures in the main text. Lemke (1998b) discusses these multi-
modal genres in detail.
The antecedents of the more common and everyday multimodal genre
of the printed magazine lie in illustrated stories and advertisements in
daily newspapers. Journalism leapt forward when photographs could be
printed on the same front page with the text of a story. Silent, captioned
film derived from slideshows with interleaved ‘titles’ for the slides, and
sound-film from an originally separate, but synchronized, musical accom-
paniment or narration. One can imagine that some of the early dioramas,
288 J. Lemke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 288)
panoramas, and cycloramas (moving panoramic paintings) must have
also been narrated as they rotated about the spectators, or vice versa.
The Gesamtkunstwerk of Wagnerian opera, a multimedia spectacle and
very nearly a sound-film without the film, defined new ways to integrate
the arts (and semiotic resources) of theater, dance, music, set design, cos-
tuming, make-up, lighting, etc.
In considering film and grand opera as multimodal genres, we can ap-
preciate even more Eisenstein’s emphasis on timing and pacing in the
construction of multimodal syntagms, whether in the sequentiality of
montage or the coordinated synchronicity of image and sound. For in
these longer works we realize that the joint-sign is only the one we appre-
hend and interpret on the shortest possible timescale, so brief that it
appears a simultaneous combination (noticing never being quite simulta-
neous). The integration of song, dance, and theatre in opera, how ever,
takes place on a much longer timescale. Song and dance participate in
the same long-term global syntagms of the artistic construction of the
whole, or even of an act within the whole, but they do not occ ur simulta-
neously and even, by convention, exclude one another. These sy ntagms
are unified across time, not within time; they are features of th e multimo-
dal genre itself.
Recent developments in multimedia semiotics allow us to examine
many of the resources available for producing syntagmatic relationship s
across di¤erent semiotic resource systems. We can compare these with
the resources that are available within more semiotically homogeneous
(though never truly unimodal) ‘texts.’
3. Some examples and approaches
In a brief discussion such as this, it is not possible to explicate the details
of the kind of multimodal analysis that social semiotics proposes. I will
instead identify some examples and cite some of the relevant literature.
Then I want to move beyond the analysis of single texts, and beyond
even our usual notions of intertextuality, to the issue of transmedia
traversals.
My own first encounter with the systematic issues of multimodal semi-
otics came in the study of scientific research articles in print media al-
ready mentioned. I described the pervasiveness of graphic figures of var-
ious kinds in these texts, and the role of captions and references to figures
that occur in the main text. In this multimodal genre the presentational
function is of prime importance, while the orientational function is re-
stricted by the conventions of the register largely to expressions of degrees
Multimodal genres 289
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 289)
of importance and degrees of certainty regarding the presentational con-
tent. The organizational function produces multimodal syntagms such as
figure-and-caption and labeled-figure. The presentational content is pre-
sented in such a way that its quantitative aspects can be represented
graphically by continuous spatial variation, linked syntagmatically to la-
bels and captions, which are then linked by cohesion to the semantemes
of the main text. Similarly, degrees of certai nty and uncertainty are typi-
cally also expressed both graphically (e.g., by ‘error bars’ on data graphs)
and verbally (e.g., by modal verbs and adverbs), again linked syn tagmati-
cally through labels on the graphs and thence cohesively to caption text
and main text. It is an interesting question whether cohesion cha ins
should in general be considered to include non-verbal graphical signs, as
clearly Galileo intended in the famous case of the intra-linear Saturn
rebus. Further discussion is o¤ered in Lemke (1998b).
The analysis of multimodal syntagms in scientific print genres led to a
further analysis of the relationships more generally between meaning by
kind (our usual categorical or ‘typological’ meaning-making by contrasts
in valeur) and meaning by degree, where quantitative rather than qualita-
tive di¤erences in signifiers function to di¤erentiate signs (a sort of ‘topo-
logical’ meaning-making). Whether we consider Saussurean signifiers and
signifieds or Peicean representamina, objects, and interpretants, all the
possible combinations of some terms of th e sign-relation being quantita-
tive versus qualitative may occur, extending our repertory of signifying
resources considerably, as natural science in particular requires, since it
seeks to connect quantitative co-variation in measures of phenomena to
linguistic explanation, often by way of a hybrid mathem atical bridge
(Lemke 1999, 2000b, 2002b).
Moving beyond the limited registers of scientific genres, I recognized
the more general importance of attitudinal meanings in the orientational
function (i.e., expressions of importance, desirability, usuality, certainty,
appropriateness, comprehensibility, and seriousness and their oppo-
sites and degrees), and undertook, following a systematic analysis for
texts of newspaper editorials (Lemke 1998c), a parallel analysis for polit-
ical cartoons (Lemke 1997). The latter has never appeared in print publi-
cation because of the di‰culty and expense of gaining permiss ion to re-
print famous, and copyrighted, cartoons. Unlike verbal text, images are
considered indivisible wholes in U.S. law, and there is no ‘fair use’ right
to reproduce them even in part for critical purposes. Nonetheless, this
study did show that once again, significant meaning-e¤ects in the car-
toons were often produced by joint multimodal syntagms of image-and-
caption, to such a degree that neither the humor, nor in some cases even
the general meaning of either the image or the caption, could be reliably
290 J. Lemke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 290)
determined without reference to the other. It is worth noting that in this
case, as for the scientific publications, the meaning relations between tex-
tual caption or label and visual image were not merely constru ed through
such organizational functions as spatial juxtaposition, but also through
semantic relations of the visual and verbal presentational content and
through coordinations (or occasionally contradictions) between the im-
plied evaluative stances and connotations regarding certainty, desirabil-
ity, etc. However conventionalized, multimodal syntagms are somewhat
more fragi le or less well determined constructs than grammatical units or
figural images on their own, and we respond by increasing the number
and functional variety of ties that bind them together. In doing so we
also create and exploit opportunities for producing more subtle inter-
semiotic meaning e¤ects, modulating the meanings of images with words,
those of words with sounds, etc.
From scientific articles and political cartoons it is a small but signifi-
cant step to the analysis of contemporary multimedia websites which
present scientific information for both technical and lay users, sometimes
in interestingly parallel formats (Lemke 2002c). Here the new resources
of interactivity enter the multimodal mix, as well as multi-cursal sequen-
tiality. In a hypertext web, there are many possible sequences, and the co-
herence of a sequence across several links depends on both thematic (pre-
sentational) cohesion and dialogic-exchange like structural relations
(orientational and organizational), as when one frame functions as a
question, and the linked frame as a response. Across many links, users/
readers generate trajectories within the website, which are themselves a
new kind of multimodal syntagm (Lemke 1998a).
But what happens when such a trajectory becomes a traversal? That is,
when it crosses between websites, across institutional, genre, and even
language and culture boundaries? Web-surfing in this sense echoes
‘channel surfing’ in television broadcast media: we shift rapidly from one
genre to another, riding the outer edge of potential cohesion and mean-
ingfulness. There is a certain pleasure in such almost-random juxta-
positions and catenations. We are stretching our penchant for making
meaningful wholes to its limit. And in so doing we are also to some extent
escaping from the institutional limitations of each separate genre, pro-
gram, or website, to make something uniquely our own and uniquely
free. I have elaborated on these themes in (Lemke 2002a, 2002c, 2003,
2005b).
Beyond such examples from my own work, there are many other im-
portant contributions to and general discussions of multimodality within
the broad framework of social semiotics (e.g., Baldry and Thibault 2005;
Kress and Van Leeuwen 2001; O’Halloran 2004; Thibault 2000). This
Multimodal genres 291
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 291)
work includes analyses of television advertisements, children’s early writ-
ing and drawing, the discourse and graphic symbolism of the mathematics
and science classroom, and much more. Space does not allow for more
extensive references to other work.
4. Traversals versus transmedia franchises
No sooner do we find, through traversals, some escape from the institu-
tional limitations on meaning of conventional genres (even multimodal
ones), than a new institutional strategy of control closely pursues us. We
do not of ten enough realize th at commercial marketing is the best-
financed source of media production in our world, and that it is often at
the cutting edge of semiotic innovation, where we ordinarily expect to
find only the arts (the least well-financed, if otherwise the freest).
If traversals represent meanings made across institutional boundaries,
across media, genres, settings, and contexts of situation, then transmedia
franchises are the response to this from innovations in marketing. Trans-
media franchises place co-branded content, and with it their ideological
messages and inducements to consumption, throughout our virtual and
spatial environment, where our individual traversals will encounter it
again and again. Among the more familiar of these franchises are those
such as the ‘Harry Potter’ and ‘StarWars’ brands. They cross both media
and genres, appearing as books, films, comics, television programming,
computer and video-games, music, merchandise, theme park rides/
attractions, and, of course, print, television, and online advertisements.
Around the time of the first release of a major new property within such
a franchise, our consumer culture environment is saturated, everywhere
we go, with their media.
What is particularly interesting for social semiotics is that all these var-
ious media and genres are making coordinated meanings. Each one is an
‘intertext’ for all the others, and none of their meanings is entirely inde-
pendent of the others. We have theories of intertextuality for linguistic
signs, but what does it mean to say that a toy, or a piece of candy, is an
intertext of a book or a film? Or to say that unique meanings, in e¤ect
new multimodal texts, generated by players of computer games in such a
transmedia franchise are intertexts of the ‘original’ books or films? We
cannot escape this dilemma by assuming that such texts remain private,
because they are now frequently recorded as digital video and shared
with others on the web, just as ‘fan fiction,’ written texts ‘in the universe
of Harry Potter6 or StarWars
TM
by ordinary readers, are systematically
shared in online communities of thousands of readers/writers.
292 J. Lemke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 292)
The goal of Hallidayan social semiotics from the beginning was to
provide tools that would enable more linguistically and semiotically in-
formed critical-ideological analysis of media (e.g., Fowler et al. 1979).
As media strategies evolve, so too must our critical-analyti cal tools, and
the theories that inform their use. Elsewhere, I have suggested that tools
such as Bakhtin’s notion of a chronot ope can be applied to understand the
semiotic contributions of pacing, place, and movement in virtual-world
media such as computer games (Lemke 2005c). I have also tried to sug-
gest that the principle of incommensurability between semiotic modalities
implies that there can never be complete consistency between images and
text, music and image, text and merchandise, game and film. There will
always be cracks in the carefully constructed and convent ionalized fa-
cades of transmedia unity, and these can be exploited for critical subver-
sion of the ideological messages that transmedia franchises often carry
(Lemke 2005a).
4.1. Social semiotics and political econom y
Social semiotics situates media in their contexts of production and circu-
lation as well as those of their use or consumption. In analyzing the
meaning-e¤ects of transmedia franchises, we need to understand the eco-
nomic imperatives of mass-markets, where there is a careful calculation of
the relative advantage of creating media that appeal to the widest possible
range of consumers versus those whose appeal to a particular market is
based on contrasting its identity and image with those of other markets.
Even core cultural identities such as those associated with ‘adolescence’
or ‘maturity,’ ‘masculinity’ or ‘femininity,’ social class, and ethnic cul-
tural heritage are today not just being manipulated by marketing media,
but are being redefined by semiotic marketing strategies in the interests of
maximizing profit (Lemke 2005a; Moje and van Helden 2005; van Hel-
den 2004). Marketing media no longer take on only the forms of the iden-
tifiable texts and genres of advertisements. All consumer prod ucts today
are marketing media, not just insofar as they are branded by logos, but
because their semiotic values are defined within systems of meaning that
have only the most tenuous connections, if any, with their material prop-
erties and use-value functions.
Semiotic valeur has thus become the lever by which we pry exchange
value loose from use value (Baudrillard 1988; Graham in press). Eco-
nomics began long ago as a general science of value (a project it has
long since given up), and we ought to remember that semiotic value is
not in fact independent of the wider general dynamics of value in our
Multimodal genres 293
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 293)
community, a dynamics which is heavily influenced today by economies
of marketing and profit. Social semiotics inherits, of necessity, the origi-
nal intellectual project of economic science: to understand how value
arises in the fusion of matter and meaning. We cannot, I believe, give a
fully persuasive account of how meaning is made unless we can say why
some meanings are made rather than others, not just individually but in-
stitutionally, and today increasingly in coordinated ways across institu-
tions. Signifiers are no longer arbitrary when they must find a place in
an already functioning semiotic ecology where the material qualities of
objects have pre-assigned relational values and meanings. Thus the ab-
stract semiotic systems of valeur relations among signs become part of a
larger system in which all possible signifiers also have relative economic
values, and these economic values are no longer merely use-values based
on their material properties, nor even exchange values based on scarcity
and demand, but semiotic values based on the cultural meanings of their
interpretants. Economic value today is created as much or more by in-
fusing matter with meaning as by shaping it for use or controlling its
availability.
Greater exchange value is assigned today to objects with no greater use
value merely by the presence of a lo go, signifying a brand, connoting a
social status. The value of the brand can even be increased by giving it
away in some material forms so that it can be sold at a higher price in
others. What matters most often is who uses it, creating an indexical con-
nection between their social status and the brands they use and display, so
that di¤erential distribution matters more than absolute scarcity. So
much is merely semiotically parasitic on existing social structures, but by
transforming use-demand into status-a spiration, the marketing economy
begins to remake the system of social identities. Increasingly, we are
what we own, not what we do. And even what we do becomes valued
not for its social usefulness but for its sign value: as an activity type, as
an occasion for the use or display of high-status consumer goods.
Among these goods today are those for which the material base is min-
imal: information in coded form, destined for decoding and display by
consumer electronics which itself conforms to the older status-goods
model. But in this mixed economy of information and tangible goods,
more and more of the economic value di¤erential between items is ac-
counted for by di¤erences in semiotic value, and those di¤erences are
more and more matters of manipulable preferences, i.e., of conformity
with personal and group identities which are themselves being produced
as part of the semiotic economy.
Information does not have the natural scarcity of matter; its marginal
cost of replication tends to zero. So what gives it its value, other than use?
294 J. Lemke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 294)
Producers today are still trying to impose artifici al scarcity, following the
model of the economics of matter (i.e., treating MP3 music files, or re-
gionally coded DVDs, as if they were artificially controlling the supply
of African diamonds). That is a failing strategy. The more successful
strategy is to di¤erentiate the market by identities and charge more for
information that appeals to the identities you have designated as having
higher status. This argument owes as much to Bourdieu (1979) as to Bau-
drillard. But there must still be some identifiable features of the informa-
tion, i.e., some semiotic features and patterns, which distinguish low-end
from high-end content. What those features are is never arbitrary for con-
tent that enters an already existing and di¤erentiated market, a material-
semiotic ecology.
The power to generate economic value and profit, the power to reshape
available social identities and the signs by which they are known, and the
power to realign the relations among these identities is political power.
More than ever before in history, semiotics is the science we need in order
to understand how political power is being re-distributed and exercised in
our times. Social semiotics is always also politi cal economy. That is what
makes it intellectually exciting as well as, hopefully, socially usefu l.
References
Baldry, A., and Thibault, P. (2005). Multimodal Transcription and Text Analysis. London:
Equinox.
Baudrillard, J. (1988). Towards a critique of the political economy of the sign. In Jean Bau-
drillard: Selected Writings, M. Poster (ed.),
PAGE NUMBERS. Stanford, CA: Stanford
University Press.
Bourdieu, P. (1979). Distinction: A Social Critique of the Judgment of Taste. Cambridge,
MA: Harvard University Press.
Eisenstein, S. (1943). The Film Sense. London: Faber.
Firth, J. R. (1968). Selected Papers of J.R. Firth 19521959. London: Longmans.
Foucault, M. (1966). The Order of Things. New York: Random House.
Fowler, R., Hodge, R., Kress, G., and Trew, T. (1979). Language and Control. London:
Routledge and Kegan Paul.
Graham, P. (in press). Hypercapitalism: New media, Language, and Social Perceptions of
Value. New York: Peter Lang.
Halliday, M. A. K. (1978). Language as Social Semiotic. London: Edward Arnold.
Helden, C. van (2004). Polyvalent meaning in multimodal youth lifestyle media. Paper pre-
sented at the First International Conference on Critical Discourse Analysis, Valencia,
Spain.
Hjelmslev, L. (1961). Prolegomena to a Theory of Language, F. Whitfield (trans.). Madison,
WI: University of Wisconsin Press.
Hodge, R. and Kress, G. (1988). Social Semiotics. Ithaca: Cornell University Press.
Kress, G. and Van Leeuwen, T. (2001). Multimodal Discourse. London: Arnold.
Multimodal genres 295
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 295)
Lemke, J. L. (1987). Strategic deployment of speech and action: A sociosemiotic analysis.
In Semiotics 1983: Proceedings of the Semiotic Society of America ‘Snowbird’ Conference,
J. Evans and J. Deely (eds.), 67–79. New York: University Press of America.
Lemke, J. L. (1997). Visual and verbal resources for evaluative meaning in political Car-
toons. Available online at http:// www-personal.umich.edu/
~
jaylemke/papers/polcart.
htm.
Lemke, J. L. (1998a). Hypertext semantics. Available online at http://www-personal.umich.
edu/
~
jaylemke/webs/hypertext/tsld001.htm.
Lemke, J. L. (1998b). Multiplying meaning: Visual and verbal semiotics in scientific text. In
Reading Science, J. R. Martin and R. Veel (eds.), 87113. London: Routledge.
Lemke, J. L. (1998c). Resources for attitudinal meaning: Evaluative orientations in text
semantics. Functions of Language 5 (1), 3356.
Lemke, J. L. (1999). Typological and topological meaning in diagnostic discourse. Discourse
Processes 27 (2), 173185.
Lemke, J. L. (2000a). Material sign processes and ecosocial organization. In Downward
Causation: Self-organization in Biology, Psychology, and Society, P. B. Andersen, C.
Emmeche, and N. O. Finnemann-Nielsen (eds.), 181213. Aarhus: Aarhus University
Press.
Lemke, J. L. (2000b). Opening up closure: Semiotics across scales. In Closure: Emergent
Organizations and their Dynamics, J. Chandler and G. van de Vijver (eds.), 100111.
New York: New York Academy of Sciences.
Lemke, J. L. (2002a). Discursive technologies and the social organization of meaning. Folia
Linguistica 35 (1–2), 7996.
Lemke, J. L. (2002b). Mathematics in the middle: Measure, picture, gesture, sign, and word.
In Educational Perspectives on Mathematics as Semiosis: From Thinking to Interpreting to
Knowing, M. Anderson et al., 215234. Ottawa: Legas.
Lemke, J. L. (2002c). Multimedia genres for science education and scientific literacy. In
Developing Advanced Literacy in First and Second Languages, M. Schleppegrell and M. C.
Colombi (eds.), 21–44. Mahwah, NJ: Erlbaum.
Lemke, J. L. (2002d). Travels in hypermodality. Visual Communication 1 (3), 299325.
Lemke, J. L. (2003). The role of texts in the technologies of social organization. In Theory
and Interdisciplinarity in Critical Discourse Analysis, R. Wodak and G. Weiss (eds.), 130
149. London: Macmillan/Palgrave.
Lemke, J. L. (2005a). Critical analysis across media: Games, franchises, and the new cultural
order. In Approaches to Critical Discourse Analysis, M. Labarta Postigo (ed.). Valencia:
University of Valencia. [CD-ROM]
Lemke, J. L. (2005b). Multimedia genres and traversals. Folia Linguistica 39 (12), 4556.
Lemke, J. L. (2005c). Place, pace, and meaning: Multimedia chronotopes. In Discourse in
Action, S. Norris and R. Jones (eds.),
PAGE NUMBERS. London: Routledge.
Malinowski, B. (1923). The problem of meaning in primitive languages. In The Meaning
of Meaning, C. K. Ogden and I. A. Richards (eds.), 296336. New York: Harcourt
Brace.
Malinowski, B. (1935). Coral Gardens and their Magic: Volume 2, The Language of Magic
and Gardening. London: George, Allen and Unwin.
Moje, E. and Helden, C. van (2005). Doing popular culture: Troubling discourses about
adolescence. In Re/Constructing ‘The Adolescent’: Sign, Symbol, and Body, J. Vadebon-
coeur and L. Stevens (eds.), 211247. New York: Peter Lang.
O’Halloran, K. (ed.) (2004). Multimodal Discourse Analysis: Systemic-Functional Perspec-
tives. London/New York: Continuum.
Thibault, P. (1991). Social Semiotics as Praxis. Minneapolis: University of Minnesota Press.
296 J. Lemke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 296)
Thibault, P. (2000). The multimodal transcription of a television advertisement: Theory and
practice. In Multimodality and multimediality in the distance learning age, A. Baldry (ed.),
331385. Campobasso: Palladino Editore.
Van Leeuwen, T. (1999). Speech, Music, Sound. New York: St. Martin’s Press.
Jay Lemke (b. aaaa) is a Professor at the University of Michigan 3jaylemke@umich.edu4.
His research interests include multimodal semiotics, social semiotics, and phenomenology of
a¤ect. His major publications include Talking Science (1990); Textual Politics (1995);
‘Across the scales of time: Artifacts, activities, and meanings in ecosocial systems’ (2000);
and ‘Travels in hypermodality’ (2002).
Multimodal genres 297
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 297)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(AutoPDF V7 26/11/08 12:02) WDG (148225mm) TimesM J-2048 Semiotica, 173 PMU:(CKN[A])26/11/2008 pp. 283–298 2048_173_12 (p. 298)
  • ... The theoretical framework of SFL has its roots in Halliday's 'language as a social semiotic' (Halliday, 1993 [1975]). In terms of these two frameworks, with SFL being largely defined "in opposition to Saussure's analytical priorities" (Chandler, 2007: 220), I am particularly interested in the concepts of choice (developed in SFL into system networks) and associative relations (Saussure's term for paradigmatic relations), both relating to the semiotic potential of resources of meaning (see also Lemke, 2009). Saussure has come under a lot of criticism from the poststructuralist school of thought that understand Saussure's ideas as positing rigid binary oppositions that function in a closed system of signification without external (social) influence. ...
    ... Consequently, we witness syntagmatic orders, that is, sequential orders (i.e. what can and cannot precede or follow a certain item), some of which are more culturally persistent (see Lemke, 2009) than others (for example, does the concept 'woman' appear in the same sequential cluster with 'powerful'). These two relational orders governing the realisation of significations are thus interrelated. ...
    Thesis
    Full-text available
    One of the growing research interests in media and Internet studies concerns how the self is constructed in the digital environment, while the complex relationship between the self and consumption continues to be of interest in consumer research. This thesis is an examination of relational being at the intersection of digital media and consumer culture. It takes a critical perspective to examine the conditions under which the contemporary self is constructed and how the self is articulated in digital contexts, and thus views the online as embedded in the offline. Rooted in social constructionism, the relational perspective sees the self as an intersection of multiple and shifting relations. The aim of the study is to gain a deeper understanding of the complexity of self-construction in our media-saturated consumer society. The study examines bloggers and fans and their mediated consumption practices through the lens of the social imaginary. The empirical material is collected from social media sites, plus-sized fashion blogs and YouTube, and analysed in the discourse analytic tradition combined with digital ethnography. The findings of the empirical studies show how ‘aspiration’ is constructed in the imaginary, with two conflicting, yet mutually constitutive notions of ‘being yourself’ and ‘improving yourself’ being negotiated at the site of the self, in the relational flow of the Internet. The studies also discuss disenfranchisement and marginalisation as properties of relationships, and show how imaginaries, in offering a range of interpretative resources for the self, also provide opportunities for counter-discourses. The study makes several theoretical and methodological contributions: within media and Internet studies, this thesis contributes to a better understanding of the embeddedness of the digital and to the ongoing discussion of how the digital is shaping the self; within consumer research, to the theorisation of relational self in the contemporary consumer context. Treating imaginaries as semiotic systems allows us to see imaginaries as constructed terrains of aspirations with complex significations. Thus, as sources of relational tension, imaginaries can be seen as implicated in the positioning, even othering, of individuals. The study suggests that the self is a fluctuating process of various alignments and disalignments within the matrix of social, cultural, and economic forces, with momentary discursive and relational achievements translating into temporary and situated congruence with others.
  • ... menu options, ad placement, and screen space occupied by videos and images (Holsanova & Nord, 2010;Kress & van Leeuwen, 1996;Tufte, 1983). The choice and balance between words and numbers-and the choice of images and video, their size and duration, nature of the content and frequency of advertisements, and multimodal rhetorical moves-shape user experience, meaning- making, and perception of truth and distant reality (Foss, 2004;Kress, 2003;Kress & Van Leeuwen, 1996; Lemke, 2009). ...
    Article
    Full-text available
    In online and video/television spaces, news media discourses incorporate multimodal design as a discursive move capable of steering meaning toward desirable implications. Around the 2016 U.S. presidential elections, while polarized news outlets made their positionality on the candidates obvious, more neutral or central news outlets revealed their preferences through subtle multimodal design choices. One of these design choices is using a quantitative visual rhetoric: persuasive multimodal moves that draw on quantification through visual, spatial, and textual manipulation-involving the choice of data representation, visual images, and illustrations, (im)balance between numeric and alphabetic texts, and general quantitative narrative. This quantitative visual rhetoric helps news outlets manipulate facts without lying with words, leaving the onus of misinterpretation on the readers/viewers. In this article, using examples from five types of media outlets-far-left, left-leaning, central, right-leaning, and far-right-we share examples of design choices we found through multimodal analysis of their quantitative visual rhetoric during the 2016 elections. We share implications for media literacy education and civic engagement.
  • ... Transmedia is a relatively new concept in media studies. It has been discussed by semioticians Scolari (2009) and Lemke (2009), sociologists Morreale and Bertone (2015) and Andò and Leonzi (2014), and media designers Phillips (2012) and Giovagnoli (2013), although perhaps the most comprehensive and significant interpretation has been by Jenkins (2006). To describe Hollywood's corporation strategies aimed at creating new nonlinear narrative trends, Jenkins wrote, A transmedia story unfolds across multiple media platforms, with each new text making a distinctive and valuable contribution to the whole. ...
  • ... The cultural category focuses on how a transmedia production fits into participatory cultures and on the effects of these productions on groups of audiences. They also investigate and elaborate the social networks of transmedia storytelling, and the audience's behavior within the field of participatory culture and engagement (Jenkins, 2006;Bolin, 2007;Dena, 2008;Evans, 2008;Perryman, 2008;Lemke, 2009;Evans, 2011;Beddows, 2012;Jensen & Vistisen, 2012;Marwick, Gray, & Ananny, 2013). The narrative category deals with the narrative part of transmedia. ...
    Article
    The area of interest is transmedia experiences in exhibitions. The research question is: How to involve visitors in a transmedia experience for an existing exhibition, which bridges the pre-, during- and post-experience? Research through design, and action research are the methods used to design and reflect on a transmedia experience for an existing exhibition. This is framed with literature about exhibitions and transmedia, and analyzed with quantitative data from a case-study of visitors in the exhibition; this is organizationally contextualized. The contribution covers a significant gap in the scientific field of designing transmedia experience in an exhibition context that links the pre- and post-activities to the actual visit (during-activities). The result of this study is a preliminary heuristic for establishing a relation between the platform and content complexity in transmedia exhibitions.
  • Chapter
    Full-text available
    BUZATO, Marcelo E. K. Data Storytelling e a dadificação de tudo: um gênero bastardo de mãe narrativa e pai banco de dados. In: LIMA-LOPES, RODRIGO ESTEVES DE; BUZATO, MARCELO EL KHOURI (Org.). . Gênero Reloading. Campinas, SP: Potes Editores, 2018. p. 95–123. Resumo A dadificação é o processo pelo qual volumes extraordinários de dados têm sido obtidos por empresas e governos via monitoramento e sensoriamento do cotidiano das pessoas. Essa massa heterogênea ganha significado (vira informação) a partir do trabalho estatístico convertido em representações visuais chamadas visualizações e/ou narrativas de dados. Neste capítulo, caracterizo as visualizações e narrativas de dados por referência aos métodos e pressupostos ontológicos e epistemológicos do quantitativismo. Após mostrar que o enquadre desses objetos semióticos nas teorias de gêneros discursivos é problemático, a despeito do seu caráter retórico e enunciativo-discursivo vir paulatinamente ganhando reconhecimento, proponho que uma forma de estudar visualizações e narrativas de dados como gêneros discursivos é caracterizá-las como gêneros híbridos de narrativa e banco de dados.
  • Article
    Full-text available
    Book of Abstracts of the 6th Critical Approaches to Discourse Analysis across Disciplines Conference (CADAAD 2016)
  • Article
    Full-text available
    Este trabalho apresenta uma proposta de (re)descrição do fenômeno do(s) (novos) letramento(s) fundamentada teórico-metodologicamente na Teoria Ator-Rede e ilustra o percurso que levou a tal proposta com uma vinheta descritiva de parte dos resultados de um estudo de cunho etnográfico que envolveu dois estudantes universitários do sudeste do Brasil por um período de dois anos. Os dados incluídos no estudo foram gerados por várias estratégias, tais como o monitoramento dos computadores pessoais dos informantes por meio de um software especializado, notas de campo, diários pessoais, observação simples e participante, além de entrevistas semiestruturadas. Objetiva convidar a comunidade de pesquisa em novos letramentos no Brasil a avaliar a utilidade de conceberem-se letramentos e subjetividades como atores-redes, assim como os limites de tal manobra teórico-metodológica. Revisa brevemente estudos sobre (novos) letramento(s) que utilizaram conceitos da Teoria Ator-Rede e conclui que a mesma ainda não foi explorada em todo o seu potencial nesse campo de pesquisa.
  • Chapter
    Jay Lemke, as a social semiotician, has been recognized as a significant and gifted researcher in linguistics, literacy, and science education.
  • Article
    This paper contributes to our understanding of learning place-based, digital literacies through urban spaces. This article explores a new analytic unit, “learning along lines,” as a tool to support the design and analysis of learning contexts where the leading mode of engagement for young learners was mobility through the city. Learning along lines emerged from a design study in which youth produced maps of their neighborhood to share with city stakeholders. Using a spatio-temporal framework, I analyze youth learning along lines they made of their neighborhood through a designed task, GPS drawing, to learn new ways of reading and writing the city with locative technologies. First, I focus on young people learning along lines they made through walking and gesture to scale their mobility to a neighborhood grid. Second, I focus on young people learning along lines they made by walking with maps and GPS devices to learn a newly mediated form of mobility. Third, I focus on youth learning along lines they made discursively to understand the narrative power and limitations of tools and maps. The analyses are intended to push our field’s understanding of digital and physical mobility in conceptualizing and designing new forms of learning locative literacies.
  • Chapter
    One of the fundamental claims of Critical Discourse Analysis (CDA) is that texts play a constitutive role in social structuration. In this chapter I will develop a theoretical perspective that tries to understand this claim in both a material and a semiotic sense. Recent research on complex systems shows that typically they are organized on and across multiple timescales. In the particular case of social-ecological systems, technologies of social organization are mediated by semiotically significant material artifacts, which persist or are functionally reproduced in long-timescale processes but are created and used on the much shorter timescales of human semiotic activity. Texts, as instances of such material-semiotic artifacts, play a key role in the organization of social systems across timescales and in the widest extension of social networks. I will argue here that this role for texts of all kinds (including those written in the grammars of architecture and bodily habitus) makes them uniquely valuable as indices of historically changing modes of social control. Extrapolating from contemporary developments in textuality (for example, hypertexts and web-surfing), I try to identify emerging forms of social control in the era of globalization. Postmodern texts and social practices already exist; they mediate new forms of social control in new ways. Progressive social projects must learn both to resist them and to make use of them, and CDA needs to extend itself in a highly interdisciplinary way in order to help us do so.
  • Article
    Full-text available
    Genres are not what they used to be. They are both more and less. More in the sense that today many genres of interest are increasingly multimodal, making their meanings through the co-deployment of resources from both language and other semiotic systems. Less in the sense that as people cross institutional and genre boundaries on shorter and shorter timescales (surfing across television channels from genre to genre, across websites from institution to institution, and living their lives between as well as within multiple jobs, tasks, and institutions), we increasingly not only hybridize formerly insulated genres, but we now also make meaning along our traversais across traditional genres. Genres are becoming units, raw material, for flexible trans-generic constructions: resources for meaning in a new, externally-oriented sense. Looking at genre from these contemporary viewpoints provides insights into the phenomenon of genre from new functional perspectives.
  • Article
    Full-text available
    That texts play a constitutive role in social structuration is one of the fundamental claims of critical discourse analysis. This article develops a theoretical perspective to understand this claim in a material as well as a semiotic sense. Recent research on complex systems shows that they are typically organized on and across multiple timescales. Texts, as material-semiotic artifacts, play a key role in the organization of social systems across timescales and in the widest extension of social networks. I propose that contemporary changes in the organization of society, such as globalization, as associated with significant shifts in the dominance of particular discursive technologies, and that hypertext and other forms of meaning making organized around traversals across traditional institutional and genre boundaries index emergent new forms of social organization and control.
  • Article
    Full-text available
    The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.