Conference Paper

Extrinsic Summarization Evaluation: A Decision Audit Task

DOI: 10.1007/978-3-540-85853-9_32 Conference: Machine Learning for Multimodal Interaction, 5th International Workshop, MLMI 2008, Utrecht, The Netherlands, September 8-10, 2008. Proceedings
Source: DBLP

ABSTRACT

In this work we describe a large-scale extrinsic evaluation of automatic speech summarization technologies for meeting speech.
The particular task is a decision audit, wherein a user must satisfy a complex information need, navigating several meetings
in order to gain an understanding of how and why a given decision was made. We compare the usefulness of extractive and abstractive
technologies in satisfying this information need, and assess the impact of automatic speech recognition (ASR) errors on user
performance. We employ several evaluation methods for participant performance, including post-questionnaire data, human subjective
and objective judgments, and an analysis of participant browsing behaviour.

Full-text preview

Available from: cstr.ed.ac.uk
  • Source
    • "In this work we conduct a user study where participants use summaries to browse meeting transcripts. Some previous work has compared extracts and abstracts for the task of a decision audit [3] , finding that human abstracts are a challenging goldstandard in terms of enabling participants to work quickly and correctly identify the relevant information. For that task, automatic extracts and the semi-automatic abstracts of Kleinbauer et al. [6] were found to be competitive with one another in terms of user satisfaction and resultant task scores. "
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper we describe a complete abstractive summarizer for meeting conversations, and evaluate the usefulness of the automatically generated abstracts in a browsing task. We contrast these abstracts with extracts for use in a meeting browser and investigate the effects of manual versus ASR transcripts on both summary types. Index Terms: summarization, automatic speech recognition, abstraction, extraction, evaluation
    Full-text · Conference Paper · Nov 2010
  • Source
    • "Hence, another trend is to use the sentences selected in the summaries as starting point for browsing the meetings. This helps users recontextualize the information and improve their ability to locate information as shown by [71]. To this end, in [69], we proposed a user interface for improving the capture of a user's information need by presenting automatically extracted keyphrases that can be refined and used to generate summaries for meeting browsing. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The CALO Meeting Assistant (MA) provides for distributed meeting capture, annotation, automatic transcription and semantic analysis of multiparty meetings, and is part of the larger CALO personal assistant system. This paper presents the CALO-MA architecture and its speech recognition and understanding components, which include real-time and offline speech transcription, dialog act segmentation and tagging, topic identification and segmentation, question-answer pair identification, action item recognition, decision extraction, and summarization.
    Full-text · Article · Sep 2010 · IEEE Transactions on Audio Speech and Language Processing
  • Source
    • "In this work we conduct a user study where participants use summaries to browse meeting transcripts . Some previous work has compared extracts and abstracts for the task of a decision audit (Murray et al., 2009) , finding that human abstracts are a challenging gold-standard in terms of enabling participants to work quickly and correctly identify the relevant information. For that task, automatic extracts and the semi-automatic abstracts of Kleinbauer et al. (2007) were found to be competitive with one another in terms of user satisfaction and resultant task scores. "
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper we present a complete sys- tem for automatically generating natural language abstracts of meeting conversa- tions. This system is comprised of com- ponents relating to interpretation of the meeting documents according to a meet- ing ontology, transformation or content selection from that source representation to a summary representation, and gener- ation of new summary text. In a forma- tive user study, we compare this approach to gold-standard human abstracts and ex- tracts to gauge the usefulness of the dif- ferent summary types for browsing meet- ing conversations. We find that our auto- matically generated summaries are ranked significantly higher than human-selected extracts on coherence and usability crite- ria. More generally, users demonstrate a strong preference for abstract-style sum- maries over extracts.
    Full-text · Conference Paper · Jun 2010
Show more