Using Static Documents as Structured and Thematic Interfaces to Multimedia Meeting Archives.

Full text

Page 1

Using Static Documents as Structured and
Thematic Interfaces to Multimedia Meeting
Archives
Denis Lalanne1, Rolf Ingold1, Didier von Rotz2, Ardhendu Behera1,
Dalila Mekhaldi1, and Andrei Popescu-Belis3
1University of Fribourg, Faculty of Science,
DIUF/DIVA, 3, ch. du Mus´ ee,
CH-1700 Fribourg, Switzerland
{denis.lalanne, rolf.ingold, ardhendu.behera,
dalila.mekaldi}@unifr.ch
2Ecole d’ing´ enieurs et d’architectes de Fribourg,
Bd de P´ erolles 80 - CP 32,
CH-1705 Fribourg, Switzerland
didier.vonrotz@eif.ch
3University of Geneva,
School of Translation and Interpreting (ETI),
TIM/ISSCO, 40, bd. du Pont d’Arve,
CH-1211 Geneva 4, Switzerland
andrei.popescu-belis@issco.unige.ch
Abstract. Static documents play a central role in multimodal applica-
tions such as meeting recording and browsing. They provide a variety of
structures, in particular thematic, for segmenting meetings, structures
that are often hard to extract from audio and video. In this article, we
present four steps for creating a strong link between static documents and
multimedia meeting archives. First, a document-centric meeting environ-
ment is introduced. Then, a document analysis tool is presented, which
builds a multi-layered representation of documents and creates indexes
that are further on used by document/speech and document/video align-
ment methods. Finally, a document-based browsing system, integrating
the various alignment results, is described along with a preliminary user
evaluation.
1 Introduction
There is a significant research trend on recording and analyzing meetings, mostly
in order to advance the research on multimodal content analysis and on multi-
media information retrieval, which are key features for designing future commu-
nication systems. Many research projects aim at archiving meeting recordings
in suitable forms for later browsing and retrieval [1,2,3,4,5,6]. However, most
of these projects do not take into account the printed documents that are often
an important part of the information available during a meeting.
S. Bengio and H. Bourlard (Eds.): MLMI 2004, LNCS 3361, pp. 87–100, 2005.
c ?Springer-Verlag Berlin Heidelberg 2005

Page 2

88D. Lalanne et al.
Printed documents have been for centuries the predominant medium of re-
mote communication between humans. With the recent advances in multimedia
and multimodal applications, new means, such as audio and video, are appearing
for exchanging information. These advances strengthen the role of printed doc-
uments, which co-exist in the physical world and the digital one. Documents are
highly thematic and structured, relatively easy to index and retrieve, and thus
can provide natural and thematic means for accessing and browsing efficiently
large multimedia meeting corpora. For that reason, it is essential to find links
between documents and multimodal annotations of meeting data such as audio
and video.
Two groups of meeting room systems emerge from a quick overview [7]. The
first group is focused on document related annotations such as handwriting and
slide analysis: Microsoft [4], FXPal [3], eClass [2], DSTC [5] and Cornell [6].
These meeting browser interfaces are based on visualizations of the slide changes
time line, and of the notes taken by participants. In these interfaces, slides and
notes are used as quick visual indexes for locating relevant meeting events and for
triggering their playback. The second group of systems is based on speech related
annotations such as the spoken word transcript: ISL [1] and eClass [2]. These
meeting browser interfaces are based on keyword search in the transcripts. In that
context, higher-level annotations such as speech acts or thematic episodes can
also be used to display quick indexes of selected meeting parts. The document-
centric and the speech-centric applications correspond respectively to the visual
and to the verbal communication modalities (or channels) of a meeting. Since
these channels are really integrated, we propose to create links between them
and include them in meeting archives and related user-interfaces. Further, we
suggest considering both the visual and the verbal links with documents in order
to fully align them with temporal data.
In this article we present four steps for bridging the gap between documents
and multimedia meeting data: document-centric meeting environment (Section
2), document recognition and indexing (Section 3), document alignment us-
ing various techniques (Sections 4, 5 and 6), and document-enabled multimedia
meeting browsing system (Section 7).
2 Document-Centric Meeting Recording
A document-centric meeting room has been installed at the University of Fri-
bourg to record different types of meetings (Figure 1). The room records several
modalities related to documents, either projected or laying on the meeting ta-
ble, and related to the participants’ discussion. The room is currently equipped
with 14 firewire webcams (8 close-ups, 6 overviews), 8 microphones, a video pro-
jector, a projection screen and a camera for capturing slides. It implements a
distributed and scalable architecture remotely controlled (6 capture boxes with
one master PC). All the capture boxes are synchronized. The meeting capture
application controls all cameras and microphones devices in the meeting room. It
enables not only basic operations like starting and stopping the recordings, but

Page 3

Using Static Documents as Structured and Thematic Interfaces 89
Fig.1. Fribourg document-centric meeting recording environment is equipped with
standard webcams and microphones for each participant and captures the document
projected and standing on the meeting table
it also enables to automate post-processing, compression and file transfers to the
server. It stands on the master PC and allows to control and to visualize all the
slave’s processing. Further, a user-friendly control interface has been developed
that allows to select which devices to use (cameras, microphones, etc), to register
the participants around the table, ant to select frame rate, resolution, etc. Post-
processing, compression, file transfer, generation of the global descriptors and a
SMIL presentation (including all the audio and video streams) are all automated
and controllable through this interface. In the future, the meeting room will be
enhanced for real-time interactions with documents, either projected or laying
on the meeting table.
Several document-centric meeting scenarios have been considered (press re-
views, lectures, reading clubs, personal presentations, etc.). In total, 22 press-
reviews, 4 job interviews and 4 student presentations have been recorded, then
manually transcribed and annotated. Further, all the meeting documents have
been indexed and structured in a canonical representation containing text, and
physical and logical structures. In the press-review scenario, participants discuss
in French the front page and the contents of one or more French-speaking news-
papers. The meetings last for about 15 minutes each. Documents’ structures
have been manually encoded in XML. Further, each meeting is accompanied by
an XML-encoded global descriptor, coming alone with audio and video files for
each participant, and both the PDF and image form of all the documents.
3Document Analysis
Documents play an important role in everyday communication. With the ever-
increasing use of the Web, a growing number of documents are published and
accessed on-line. Unfortunately, document structures are not often considered,
which considerably weaken users’ browsing and searching experience. There are
many levels of abstraction in a document, conveyed by its various structures:
thematic, physical, logical, relational or even temporal. In most of the search

Page 4

90D. Lalanne et al.
engines and information retrieval systems, this multi-layered structure is not
taken into account; documents are indexed in the best case according to their
thematic structure or simply represented as a bag of words. The form of the
documents, i.e. their layout and logical structures, is underestimated and could
carry important clues about how the document is organized. We believe that
document structure extraction will drastically improve documents indexing and
retrieval, as well as linking with other media.
We have chosen to analyze PDF documents mainly because PDF has become
the common format for exchanging printable documents and because it preserves
the display format. The use of PDF is frequently limited to displaying and print-
ing, regardless of the improvements it could bring to search and retrieval. We
believe that the extraction from documents of both layout and logical structure
will enrich PDF indexing and linking with other media. In particular, in the
present application, the document structures allow the linking of PDF docu-
ments with the speech transcript. We recently proposed a novel approach that
merges low-level extraction methods applied on PDF files with layout analysis
of a synthetically generated TIFF image [8].
In the field of document analysis, image segmentation aims at zoning a given
document image into homogenous regions that have meaningful properties, e.g.
text, image, graphics, etc. Our segmentation algorithm first extracts threads,
frames and text lines, then separates image and text zones, and finally merges
lines into homogeneous blocs. The algorithm’s input is the TIFF image generated
from the PDF file, while the output is an XML file, which describes the segmen-
tation results for the document components mentioned above. In parallel, the
various objects contained in the PDF file, including text, images, and graphics,
are extracted. The PDF file is first disambiguated; the different representations
are then homogenized, and the cleaned PDF is parsed into a unique tree, which
can be then transformed either into an XML document, e.g. in SVG. Finally,
the objects extracted from the PDF document are matched with the result of
the layout analysis in order to construct a structured XML representation of the
PDF document. For example the text is matched with the physical blocks in
order to create associations between the two [8].
In order to produce a proper ground-truth for our press reviews, documents
have been segmented manually [9]. The PDF documents corresponding to the
newspapers’ front pages discussed in the recorded meetings have been first con-
verted automatically to text and then logically structured in XML along with
information about the layout structure, i.e. the bounding boxes of each logi-
cal block, topological positions, fonts, etc. For instance, a Newspaper front page
bears the newspaper’s Name, the Date, one MasterArticle, zero, one or more High-
lights, one or more Articles, etc. Each Article has a Title, Authors, a Content, etc.
4Temporal Document Alignments
In order to browse multimedia corpuses through static documents, it is first
necessary to build links between those documents, which are non-temporal, and

Page 5

Using Static Documents as Structured and Thematic Interfaces91
other media, which are generally temporal. We call “document temporal align-
ment” the operation of extracting the relationships between a document excerpt,
at variable granularity levels, and the meeting presentation time. Document tem-
poral alignment create links between document extracts and the time intervals
in which they were in: (a) the speech focus, (b) the visual focus and/or into (c)
the gestural focus of a meeting. It is thus possible to align document extracts
with audio and video extracts, and by extension with any annotation of audio
and/or video and/or gesture. There are three modalities related to documents
that we use for further temporal alignment:
1. Speech: documents’ content is matched with speech transcripts’ content,
which holds timestamps for each speaker turn and speech utterance.
2. Video: electronic documents are matched with extracted frames from the
meeting’s documents videos (e.g. slides projected) in order to extract time
stamps associated with visible state changes.
3. Gesture: gestural interactions with documents are captured and analyzed
(e.g. pointing a document), in order to find out when and which specific
document part was in gestural focus.
Both document/video and document/gesture alignment are video-based and
bridge the gap between document excerpts and video extracts. These approaches
take advantage of the observable events related to documents that are visible dur-
ing meetings, such as projected documents or documents standing on the meeting
table. Documents’ intra-events (slide changes, animations, zooming, scrolling,
etc.) are handled by the document/video alignment, whereas documents’ extra-
events (e.g. pointing a projected document with a laser-beam, finger-pointing
of documents laying on a table, pen-based gestural interactions on a TabletPC,
etc.) are handled by the document/gesture alignment, in order to find out when
and which specific document part was in gestural focus.
In the next two sections, the advancement of our work on document/speech
and document video alignments is presented. Document/gesture has not yet been
handled; the technique we envision to use to solve this alignment combines two
established domains: gestural interaction and document analysis. The gestural
analysis leads to high-level annotations on gests (such as pointing, circling, un-
derlying, etc.) with their associated timestamps. Document analysis techniques
provide methods for extracting the logical structure from electronic documents,
as described in Section 3, which will greatly help to determine which document
block has been pointed, circled or underlined. Gestural interactions with docu-
ments have been rarely tackled and should lead to new document related anno-
tations and also to real-time prototypes that bring back old technologies such as
paper documents to the digital world [10,11]. In general, this document/gesture
alignment will help answering two questions:
1. When was a document pointed to?
2. Which document or document part was pointed to?