Conference Paper

Creative Help: A Story Writing Assistant

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

We present Creative Help, an application that helps writers by generating suggestions for the next sentence in a story as it being written. Users can modify or delete suggestions according to their own vision of the unfolding narrative. The application tracks users’ changes to suggestions in order to measure their perceived helpfulness to the story, with fewer edits indicating more helpful suggestions. We demonstrate how the edit distance between a suggestion and its resulting modification can be used to comparatively evaluate different models for generating suggestions. We describe a generation model that uses case-based reasoning to find relevant suggestions from a large corpus of stories. The application shows that this model generates suggestions that are more helpful than randomly selected suggestions at a level of marginal statistical significance. By giving users control over the generated content, Creative Help provides a new opportunity in open-domain interactive storytelling.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... In recent years, the field of NLG has progressed by leaps and bounds due to the development of neural language models capable of learning the structure of language by ingesting billions of written words (Chowdhery et al., 2022;Brown et al., 2020). There has been considerable work in applying these advancements toward the development of AI-powered tools for creative writing, but nearly all previous research in this space has evaluated their methods either with amateur writers or with crowd workers paid to assess performance on narrowly defined tasks (Clark et al., 2018;Roemmele and Gordon, 2015;Nichols et al., 2020). While these sorts of evaluations are valuable as preliminary assessments, we believe it is also crucial to solicit feedback from actual domain experts in creative writing: professional writers, educators, and language experts. ...
... More recent work took a data-driven approach, consulting a knowledge base of commonsense assertions or world facts to improve story coherence (Liu and Singh, 2002;McIntyre and Lapata, 2009). Among recent work, some have strived to generate entire stories without human intervention (Fan et al., 2018) while others have stressed the importance of designing AI systems that prioritize human involvement in the story creation process (Riedl and Young, 2006;Swanson et al., 2021;Akoury et al., 2020;Roemmele and Gordon, 2015). With Wordcraft, we take the latter position, focusing on the potential of story writing AI to be useful tool for writers, just as a traditional word processor, a spellchecker, or a deck of ideation cards are tools to expedite the writing process. ...
... Most past user studies involving these story-writing AI have conducted evaluation in contrived settings testing narrow functionality, typically with amateur writers (Clark et al., 2018;Roemmele and Gordon, 2015;Roemmele, 2021;Nichols et al., 2020). One notable exception is the work of Akoury et al. (2020), who incorporated a suggestion engine into an online story writing game and analyzed how game users interacted with it. ...
Preprint
Recent developments in natural language generation (NLG) using neural language models have brought us closer than ever to the goal of building AI-powered creative writing tools. However, most prior work on human-AI collaboration in the creative writing domain has evaluated new systems with amateur writers, typically in contrived user studies of limited scope. In this work, we commissioned 13 professional, published writers from a diverse set of creative writing backgrounds to craft stories using Wordcraft, a text editor with built-in AI-powered writing assistance tools. Using interviews and participant journals, we discuss the potential of NLG to have significant impact in the creative writing domain--especially with respect to brainstorming, generation of story details, world-building, and research assistance. Experienced writers, more so than amateurs, typically have well-developed systems and methodologies for writing, as well as distinctive voices and target audiences. Our work highlights the challenges in building for these writers; NLG technologies struggle to preserve style and authorial voice, and they lack deep understanding of story contents. In order for AI-powered writing assistants to realize their full potential, it is essential that they take into account the diverse goals and expertise of human writers.
... These existing MICIs have shown promise in several ways. In particular, MICIs that function by providing shortterm story continuations have proven effective at suggesting viable next steps for a story (Roemmele and Gordon 2015); taking the story in unexpected directions (Kreminski et al. 2020a;Calderwood et al. 2020;Singh et al. 2022); and creating a sense of shared authorship (Samuel 2016) between the user and system (Kreminski et al. 2020a;Calderwood et al. 2020;Singh et al. 2022). ...
... All rights reserved. they have a tendency to pull the story in unwanted directions (Roemmele and Gordon 2015;Calderwood et al. 2020;Singh et al. 2022) or to otherwise create a sense of long-term directionlessness (Kreminski et al. 2020a) that inhibits the development of coherent high-level story structure. ...
... Beyond plot event-based systems such as Writing Buddy and Why Are We Like This?, a number of attempts have also been made to facilitate mixed-initative storytelling by providing continuation suggestions in the form of unstructured prose. Early examples of this approach can be found in the Say Anything (Swanson and Gordon 2012) and Creative Help (Roemmele and Gordon 2015) systems, which use case-based reasoning to find sentences similar to the user's most recently typed sentence in a large database of preauthored stories, then suggest these sentences as continu-ations. More recently, textual continuations provided by language models have been used to support storytelling in a relatively unmediated way (Manjavacas et al. 2017;Calderwood et al. 2020). ...
Article
Full-text available
We present Loose Ends, a mixed-initiative co-creative storytelling play experience in which a human player and an AI system work together to compose a story. Loose Ends specifically aims to provide computational support for managing multiple parallel plot threads and bringing these threads to satisfying conclusions—something that has proven difficult in past attempts to facilitate playful mixed-initiative storytelling. We describe the overall human-AI interaction loop in Loose Ends, including the implementation of the rules-based AI system that enables this interaction loop; discuss four examples of desirable mixed-initiative interactions that are possible in Loose Ends, but not in similar systems; and present results from a preliminary expert evaluation of Loose Ends. Altogether, we find that Loose Ends shows promise for creating a sense of coauthorship in the player while also mitigating the directionlessness reported by players of earlier systems.
... Prior work shows that exposing authors to a collaborator that provides independent suggestions can spark new ideas (Garfield, 2008). This has motivated a line of work in machine-in-the-loop writing (Clark et al., 2018;Roemmele and Gordon, 2015;Samuel et al., 2016) where a human collaborates with a model to complete a writing task. However, recent work has shown that providing humans a draft generated by a machine is not very helpful because it may 1 Our code and pretrained models are available at https://github.com/vishakhpk/ ...
... In this work, we aim to provide a form of interaction that gives human authors more control over the content while also assisting them to better express their own ideas (Roemmele and Gordon, 2015). We focus on the setting where authors have a clear writing outline but would benefit from suggestions on wording or framing. ...
... WordNet example sentences Words that elicit emotion I attacked the problem as soon as I was up. Gordon et al. (2015) Text collected by Mohler et al. (2015) Metaphors in text I will be out in the city today, feeling the vinous veinous thrust of blood, the apple-red circulation of democracy, its carnal knowledge without wisdom. Bostan et al. (2020) Headlines Textual cues associated with emotion Detention centers will shock the conscience of the nation. ...
... Prior work shows that exposing authors to a collaborator that provides independent suggestions can spark new ideas (Garfield, 2008). This has motivated a line of work in machine-in-the-loop writing (Clark et al., 2018;Roemmele and Gordon, 2015;Samuel et al., 2016) where a human collaborates with a model to complete a writing task. However, recent work (Akoury et al., 2020;Clark et al., 2018) has shown that providing humans a draft generated by a machine is not very helpful because it may diverge from the direction envisioned by the author. ...
... As a result, very little machine-generated text is ultimately retained. In this work, we aim to provide a form of interaction that gives human authors more control over the content while also assisting them to better express their own ideas (Roemmele and Gordon, 2015). ...
... Which users find CRA more helpful? Our main hypothesis is that CRA benefits human authors by giving them more control over the global content (Roemmele and Gordon, 2015) and providing local wording suggestions. Thus, its effectiveness relies on the assumption that the user has a coherent writing plan, which may or may not be true depending on the skill level of the writer. ...
Preprint
Machine-in-the-loop writing aims to enable humans to collaborate with models to complete their writing tasks more effectively. Prior work has found that providing humans a machine-written draft or sentence-level continuations has limited success since the generated text tends to deviate from humans' intention. To allow the user to retain control over the content, we train a rewriting model that, when prompted, modifies specified spans of text within the user's original draft to introduce descriptive and figurative elements locally in the text. We evaluate the model on its ability to collaborate with humans on the task of creative image captioning. On a user study through Amazon Mechanical Turk, our model is rated to be more helpful than a baseline infilling language model. In addition, third-party evaluation shows that users write more descriptive and figurative captions when collaborating with our model compared to completing the task alone.
... Unlike more constrained text generation tasks, such as translation or summarization, fiction writing allows for almost infinite creative freedom, which budding authors often find cognitively overwhelming (Rose, 1980). Machine-in-the-loop storytelling (Clark et al., 2018), in which an author obtains automatically generated sentences or paragraphs when stuck with writer's block, lowers the barrier to entry for creative writing (Roemmele and Gordon, 2015). To spur research in this area, we partner with STO-RIUM, 1 an online collaborative storytelling platform, to introduce a new dataset and evaluation methodology for story generation. ...
... Our work connects more closely to existing machine-in-the-loop storytelling work (Roemmele and Gordon, 2015;Samuel et al., 2016;Clark et al., 2018), in which systems work in concert with users to collaboratively author a narrative. Much like the Creative Help platform of Roemmele and Gordon (2015), we provide writing assistance by interactively generating continuations of STORIUM stories. ...
... Our work connects more closely to existing machine-in-the-loop storytelling work (Roemmele and Gordon, 2015;Samuel et al., 2016;Clark et al., 2018), in which systems work in concert with users to collaboratively author a narrative. Much like the Creative Help platform of Roemmele and Gordon (2015), we provide writing assistance by interactively generating continuations of STORIUM stories. We improve over Roemmele and Gordon (2015) by evaluating a trained model (instead of a retrievalbased approach) with a large user population. ...
... Unlike more constrained text generation tasks, such as translation or summarization, fiction writing allows for almost infinite creative freedom, which budding authors often find cognitively overwhelming (Rose, 1980). Machinein-the-loop storytelling (Clark et al., 2018), in which an author obtains automatically generated sentences or paragraphs when stuck with writer's block, lowers the barrier to entry for creative writing (Roemmele and Gordon, 2015). To spur research in this area, we partner with STORIUM, 1 an online collaborative storytelling platform, to introduce a new dataset and evaluation methodology for story generation. ...
... Our work connects more closely to existing machine-in-the-loop storytelling work (Roemmele and Gordon, 2015;Samuel et al., 2016;Clark et al., 2018), in which systems work in concert with users to collaboratively author a narrative. Much like the Creative Help platform of Roemmele and Gordon (2015), we provide writing assistance by interactively generating continuations of STORIUM stories. ...
... Our work connects more closely to existing machine-in-the-loop storytelling work (Roemmele and Gordon, 2015;Samuel et al., 2016;Clark et al., 2018), in which systems work in concert with users to collaboratively author a narrative. Much like the Creative Help platform of Roemmele and Gordon (2015), we provide writing assistance by interactively generating continuations of STORIUM stories. We improve over Roemmele and Gordon (2015) by evaluating a trained model (instead of a retrievalbased approach) with a large user population. ...
Preprint
Systems for story generation are asked to produce plausible and enjoyable stories given an input context. This task is underspecified, as a vast number of diverse stories can originate from a single input. The large output space makes it difficult to build and evaluate story generation models, as (1) existing datasets lack rich enough contexts to meaningfully guide models, and (2) existing evaluations (both crowdsourced and automatic) are unreliable for assessing long-form creative text. To address these issues, we introduce a dataset and evaluation platform built from STORIUM, an online collaborative storytelling community. Our author-generated dataset contains 6K lengthy stories (125M tokens) with fine-grained natural language annotations (e.g., character goals and attributes) interspersed throughout each narrative, forming a robust source for guiding models. We evaluate language models fine-tuned on our dataset by integrating them onto STORIUM, where real authors can query a model for suggested story continuations and then edit them. Automatic metrics computed over these edits correlate well with both user ratings of generated stories and qualitative feedback from semi-structured user interviews. We release both the STORIUM dataset and evaluation platform to spur more principled research into story generation.
... One such application focuses on providing automated support to human authors for story writing. In particular, Roemmele and Gordon (2015), Khalifa et al. (2017), Manjavacas et al. (2017), and Clark et al. (2018) have developed systems that automatically generate suggestions for new sentences to continue an ongoing story. ...
... In this work, we analyze a set of user interactions with the application Creative Help (Roemmele and Gordon, 2015), where users make 'help' requests to automatically suggest new sentences in a story, which they can then freely modify. We take advantage of Creative Help's functionality that tracks authors' edits to generated sentences, resulting in an alignment between each original suggestion and its modified form. ...
... Once one minute has elapsed since the author last edited the sentence, the application logs the modified sentence alongside its original version. See Roemmele and Gordon (2015) for further details about this tracking and logging functionality. The result of authors' interactions with the application is a dataset aligning generated suggestions to their corresponding modifications along with the story context that precedes the help request. ...
... One line of research in interactive storytelling aims to construct computational systems that assist the human interactor in making up a story of their own [9]for instance by providing the interactor with a storytelling partner in the form of an artificially intelligent storytelling system, resulting in a mixed-initiative cocreative [10] approach to storytelling. Systems like Say Anything [13], Creative Help [11], and TaleBrush [1] enable collaborative human/AI storytelling at the level of the prose that constitutes a written story, while systems like Writing Buddy [12] and Why Are We Like This? [6,5] enable collaborative storytelling at the level of the plot events that constitute an abstract narrative structure. ...
... Loose Ends ( Figure 1) is a mixed-initiative creative interface [2] for playful storytelling. Much like several previous systems in this area of research [13,11,12,6,1], Loose Ends is an interactive system that assists users in producing non-interactive stories. We specifically conceive of Loose Ends as an AI-based narrative instrument [9]: a system that can be played to produce narrative, in much the same way that a musical instrument can be played to produce music. ...
Chapter
Full-text available
We present a demonstration of Loose Ends, a mixed-initiative creative interface for playful storytelling that assists players in managing plot threads to achieve storytelling goals related to high-level story structure. From a design perspective, Loose Ends is an example of a narrative instrument: an expression-oriented playable system that can be played to produce narrative, in much the same way that musical instruments are played to produce music.KeywordsNarrative instrumentsMixed-initiative co-creativityInteractive emergent narrativeStory sifting
... logical condition in which the writer experiences an inhibition in creativity and lack of new ideas while writing a story. Although some computational methods have been developed to assist writers in this condition by suggesting what to write next (Roemmele and Gordon 2015), the use of deep learning for this purpose has only been proposed but never successfully tested (Roemmele 2016). ...
... Previous work that constitutes the current state of the art only regards the very last written sentence as a context for its suggestion and only suggests whole sentences that already exist in its database of stories (Roemmele and Gordon 2015). Our work improves on this method by conditioning on the whole written story and sampling the proposed continuations from a learned generative model. ...
Article
We present a novel approach to modeling stories using recurrent neural networks. Different story features are extracted using natural language processing techniques and used to encode the stories as sequences. These sequences can be learned by deep neural networks, in order to predict the next story events. The predictions can be used as an inspiration for writers who experience a writer's block. We further assist writers in their creative process by generating visualizations of the character interactions in the story. We show that suggestions from our model are rated as highly as the real scenes from a set of films and that our visualizations can help people in gaining deeper story understanding.
... Interactive writing assistants have wide applications in creative writing (Roemmele and Gordon, 2015;Clark et al., 2018;Akoury et al., 2020), education (Luo et al., 2015), and gaming (Walton, 2020). Nevertheless, the existing systems' options usually do not provide fine-grained control and/or ...
... In some prior work (Keskar et al., 2019;Tu et al., 2019), users choose among a static set of predefined attributes (e.g., sentiment) that only provide coarse-grained control. Other work (Roemmele and Gordon, 2015;Clark et al., 2018) presents users with multiple generated continuations, which requires substantial reading effort and might not contain topics that users want to see. Finally, options could be nodes in a plot graph that are handcrafted (Luo et al., 2015) or derived from a collaboration between humans and machine (Li et al., 2013), but such choices are usually limited due to the high cost of preparing the options. ...
... Typically, writers rate a single system's quality after some period of use, for example while authoring an entire story or poem (e.g., Clark et al., 2018;Ghazvininejad et al., 2017). A model's quality is measured using Likert scale scores, sometimes combined with additional analysis, like the type or quantity of writer edits (e.g., Roemmele and Gordon, 2015;Akoury et al., 2020). ...
... Collaborative writing systems have been developed in domains like poetry (Ghazvininejad et al., 2017), slogans (Clark et al., 2018, and stories (Roemmele and Gordon, 2015;Goldfarb-Tarrant et al., 2019;Akoury et al., 2020). Like Storium (Akoury et al., 2020), we focus on the potential to use these systems as evaluation platforms. ...
... Most of them focused on lower-level text generation or proofreading. For example, the Creative Help system used a recurrent neural network model to generate suggestions for creative writing [34]. The Scheherazade system was developed for interactive narrative generation [24]. ...
... Data Preparation: In response to the need of knowing where the writer actually got stuck, we acquired the data collected by the Creative Help system [34] for further study. Creative Help is an online writing application where users can freely write stories. ...
Preprint
Ideation is essential for creative writing. Many authors struggle to come up with ideas throughout the writing process, yet modern writing tools fail to provide on-the-spot assistance for writers when they get stuck. This paper introduces Heteroglossia, an add-on for Google Docs that allows writers to elicit story ideas from the online crowd using their text editors. Writers can share snippets of their working drafts and ask the crowd to provide follow-up story ideas based on it. Heteroglossia employs a strategy called "role play", where each worker is assigned a fictional character in a story and asked to brainstorm plot ideas from that character's perspective. Our deployment with two experienced story writers shows that Heteroglossia is easy to use and can generate interesting ideas. Heteroglossia allows us to gain insight into how future technologies can be developed to support ideation in creative writing
... Most story generation research has focused on modeling the plot, characters, and primary action of the story, using simplistic methods for producing the actual linguistic form of the stories (Turner, 1993;Riedl and Young, 2010). More recent work learns from data how to generate stories holistically without a clear separation between content selection and surface realization (McIntyre and Lapata, 2009), with a few recent methods based on recurrent neural networks (Roemmele and Gordon, 2015;Huang et al., 2016). ...
... More recent work in story generation has focused on data-driven methods (McIntyre and La pata, 2009McIntyre, 2011;Elson, 2012;Daza et al., 2016;Roemmele, 2016). The generation problem is often constrained via anchoring to some other input, such as a topic or list of keywords (McIntyre and Lapata, 2009), a sequence of images (Huang et al., 2016), a set of loosely-connected sentences (Jain et al., 2017), or settings in which a user and agent take turns adding sentences to a story (Swanson and Gordon, 2012;Roemmele and Gordon, 2015;Roemmele, 2016). ...
... As in mixed-initiative interface work, the goal of our work is to explore interaction paradigms and to combine human and computational strengths to enhance human ability [1]. The mixed-initiative setup has been used for creative tasks such as game design [34], and adapting Several tools have been developed to provide suggestions to assist people in writing, both within the research community [30,26] and as personal projects [27]. Swanson and Gordon's "Say Anything" [30] provides suggestions for writing short stories by prompting writers with full sentences retrieved from a database of stories scraped from the web after every turn of writing. ...
... Swanson and Gordon's "Say Anything" [30] provides suggestions for writing short stories by prompting writers with full sentences retrieved from a database of stories scraped from the web after every turn of writing. In "Creative Help" [26], writers are offered suggestions as they write stories, but only when they explicitly request them (i.e., a pull method of interaction). While these systems retrieve their sentences from existing stories, we use natural language generation to provide suggestions. ...
Conference Paper
As the quality of natural language generated by artificial intelligence systems improves, writing interfaces can support interventions beyond grammar-checking and spell-checking, such as suggesting content to spark new ideas. To explore the possibility of machine-in-the-loop creative writing, we performed two case studies using two system prototypes, one for short story writing and one for slogan writing. Participants in our studies were asked to write with a machine in the loop or alone (control condition). They assessed their writing and experience through surveys and an open-ended interview. We collected additional assessments of the writing from Amazon Mechanical Turk crowdworkers. Our findings indicate that participants found the process fun and helpful and could envision use cases for future systems. At the same time, machine suggestions do not necessarily lead to better written artifacts. We therefore suggest novel natural language models and design choices that may better support creative writing.
... At the intersection between natural language generation, computational creativity, and human-computer interaction research is the vision of automated tools that collaborate with people in authoring creative text. The recent application Creative Help [4] explores this vision for story writing. The interface is simple: authors type \help\ to generate a suggestion for a new sentence in an ongoing story, which they can then edit. ...
... Figure 1 shows an example with the suggestion returned by the help request underlined. In [4], a nearest-neighbors similarity approach was used to produce Creative Help suggestions by retrieving sentences from a large story corpus. The current demo applies a different generation approach that dynamically generates novel sentences word-byword. ...
Conference Paper
Full-text available
This work demonstrates an interface, Creative Help, that assists people with creative writing by automatically suggesting new sentences in a story. Authors can freely edit the generated suggestions, and the application tracks their modifications. We make use of a Recurrent Neural Network language model to generate suggestions in a simple probabilistic way. Motivated by the theorized role of unpredictability in creativity, we vary the degree of randomness in the probability distribution used to generate the sentences, and find that authors' interactions with the suggestions are influenced by this randomness.
... /feduc. . Artificial intelligence may point a way forward by monitoring and enhancing the creative process in students without putting additional workload on teachers (Swanson and Gordon, 2012;Muldner and Burleson, 2015;Roemmele and Gordon, 2015;Clark et al., 2018;Kovalkov et al., 2020;Beaty and Johnson, 2021). However, for such systems to be successful, we require a model of creativity that can be implemented computationally (Kovalkov et al., 2020). ...
Article
Full-text available
This paper provides a review of the existing literature on creativity. More importantly, the review is distilled into a novel, graph-based model of creativity with three target audiences: Educators, to gain a concise overview of the research and theory of creativity; educational researchers, to use the interactions predicted by theory to guide experimental design; and artificial intelligence researchers, who may use parts of the model as a starting point for tools which measure and facilitate creativity.
... The key challenge in collaborative writing is to understand user intent so as to provide timely and useful suggestions. Prior work in story writing (Roemmele and Gordon, 2015;Clark et al., 2018) presented sentence-level continuations at locations specified by a user. Akoury et al. (2020); Lee et al. (2022) took this a step further providing users with a paragraph of text which they could further edit in story writing and argumentative writing tasks. ...
Preprint
Recent work in training large language models (LLMs) to follow natural language instructions has opened up exciting opportunities for natural language interface design. Building on the prior success of LLMs in the realm of computer-assisted creativity, we aim to study if LLMs can improve the quality of user-generated content through collaboration. We present CoPoet, a collaborative poetry writing system. In contrast to auto-completing a user's text, CoPoet is controlled by user instructions that specify the attributes of the desired text, such as Write a sentence about `love' or Write a sentence ending in `fly'. The core component of our system is a language model fine-tuned on a diverse collection of instructions for poetry writing. Our model is not only competitive with publicly available LLMs trained on instructions (InstructGPT), but is also capable of satisfying unseen compositional instructions. A study with 15 qualified crowdworkers shows that users successfully write poems with CoPoet on diverse topics ranging from Monarchy to Climate change. Further, the collaboratively written poems are preferred by third-party evaluators over those written without the system.
... To inspire, however, researchers take a careful attitude towards keeping the autonomous role of human writers in the collaboration. Suggestions such as 'Machine in the loop' [15], 'Say Anything' [56] and 'Creative Help' [50] are provided so that when human get inspired they still have the final control over the final result. Osone et al. [44] are more cautious, who focus on improvement based on writer's written content and further increase their interest in creating more new ideas on their own. ...
Preprint
Full-text available
The creation process of design fiction is going participatory and inclusive with non experts. Recognizing the potential of artificial intelligence in creativity support, we explore the use of AI assistance in creating design fiction. This investigation is based on a workshop on future work in 2040 with Chinese youth. We look into fiction quality, participants experiences with the AI agent, and their ways of incorporating those texts into writing. Our findings show that human writers while responding to messy and unexpected AI-generated texts, can elevate the richness and creativity in writing and initiate joyful and inspirational interactions. Furthermore, for the design of AI assistance in creativity support, we suggest two implications of enhancing interactional quality between human and AI and prompt programming. Our study indicates the potential of applying design fiction outside the design context using a more inclusive approach for future speculation with critical reflection on technology.
... A word-based score treats the response and the reference as a bag-of-words and does not usually consider the context of an input text for the evaluation. Instead, it compares and evaluates the response against the available reference via a set of heuristic features, such as Precision or Recall, and computes the similarity between text units based on the matching overlap, or based on the number of word edits required to make the response similar to the given reference [173,174]. The word-based scores include a wide range of error scores, such as BLEU [175], METEOR [176], ROUGE [177], NIST [178] and CIDEr [179]. ...
Preprint
Full-text available
Question answering (QA) systems are among the most important and rapidly developing research topics in natural language processing (NLP). A reason, therefore, is that a QA system allows humans to interact more naturally with a machine, e.g., via a virtual assistant or search engine. In the last decades, many QA systems have been proposed to address the requirements of different question-answering tasks. Furthermore, many error scores have been introduced, e.g., based on n-gram matching, word embeddings, or contextual embeddings to measure the performance of a QA system. This survey attempts to provide a systematic overview of the general framework of QA, QA paradigms, benchmark datasets, and assessment techniques for a quantitative evaluation of QA systems. The latter is particularly important because not only is the construction of a QA system complex but also its evaluation. We hypothesize that a reason, therefore, is that the quantitative formalization of human judgment is an open problem.
... Story generation is the challenging problem of selecting or completing a sequence of events that can be interpreted as a story. Stories in the TaleMaker's database can be accessed by story generators, such as Say Anything and Creative Help, since these systems require a large collection of stories to propose the next plot point (or sentence) for a story [10,11]. Each story in our database contains multiple plot points per turn, both the selected one and the ones not selected, along with the number of votes assigned to each as relevance information. ...
Conference Paper
Full-text available
Several datasets of stories and text have been proven useful for a variety of research fields. Yet, many of these datasets have suffered from the burden of being manually authored and/or annotated, affecting their size and potential to grow. To overcome this problem, we propose a novel database of stories collected from TaleMaker, an online multiplayer game that facilitates the playful co-creation of a story in order to eliminate the tedious task of authoring and annotating a dataset of stories. TaleMaker's database relational schema provides a simple story representation, in which stories are named and clearly annotated. A story is composed of a sequence of plot points, each with several slots (e.g. action, character, location) filled with sense-annotated tokens (words) associated with a WordNet synset. In this paper, we describe in detail the database schema of TaleMaker's stories repository. In addition, we suggest some of the potential applications of this repository of stories, including fostering research in fields such as story generation, narrative world generation, and word sense disambiguation. CCS CONCEPTS • Information systems → Relational database model; • Human-centered computing → Collaborative and social computing theory, concepts and paradigms.
... This thesis envisions the task of narrative auto-completion applied to helping an author write a story. My advisor and I have built an application called Creative Help that provides this type of automated writing assistance (Roemmele and Gordon 2015). In this application, a user writes a story and when she types \help\, the application returns a suggested next sentence which the user can edit like any other text in the story. ...
Article
This thesis explores the use of a recurrent neural network model for a novel story generation task. In this task, the model analyzes an ongoing story and generates a sentence that continues the story.
... Related, but not using summarization, are the many tools that support creating text; this is only a small overview: Already in 1982, Macdonald et al. [27] built a "writer's workbench", which could comment on stylistic features. Recent work typically generate text suggestions (e.g. for the next sentence [38] or paragraph [54] or in a sidebar [42]). More broadly, this co-creation through interleaved human writing and AI-generated text is a common approach (e.g. ...
Preprint
We propose a text editor to help users plan, structure and reflect on their writing process. It provides continuously updated paragraph-wise summaries as margin annotations, using automatic text summarization. Summary levels range from full text, to selected (central) sentences, down to a collection of keywords. To understand how users interact with this system during writing, we conducted two user studies (N=4 and N=8) in which people wrote analytic essays about a given topic and article. As a key finding, the summaries gave users an external perspective on their writing and helped them to revise the content and scope of their drafted paragraphs. People further used the tool to quickly gain an overview of the text and developed strategies to integrate insights from the automated summaries. More broadly, this work explores and highlights the value of designing AI tools for writers, with Natural Language Processing (NLP) capabilities that go beyond direct text generation and correction.
... For example, the aforementioned AI text suggestions may not only improve text input, but can also provide incentives for exploring new wording or topics. There are several papers about the use of suggestions in creative writing [9,31]. The work by Arnold et al. [3] shows that typing with suggestions impact the writing and results in a shorter and more predictable text. ...
Preprint
Neural language models have the potential to support human writing. However, questions remain on their integration and influence on writing and output. To address this, we designed and compared two user interfaces for writing with AI on mobile devices, which manipulate levels of initiative and control: 1) Writing with continuously generated text, the AI adds text word-by-word and user steers. 2) Writing with suggestions, the AI suggests phrases and user selects from a list. In a supervised online study (N=18), participants used these prototypes and a baseline without AI. We collected touch interactions, ratings on inspiration and authorship, and interview data. With AI suggestions, people wrote less actively, yet felt they were the author. Continuously generated text reduced this perceived authorship, yet increased editing behavior. In both designs, AI increased text length and was perceived to influence wording. Our findings add new empirical evidence on the impact of UI design decisions on user experience and output with co-creative systems.
... Creative Help works on the same principle as Say Anything, however, users have the option of modifying or removing suggestions within the story as they see fit [21]. These tools inspired Writing Buddy, a playful, mixed-initiative tool that integrates design principles of casual creators [6], and uses a social simulation engine called Ensemble to help authors create story beats and actions for their stories [23]. ...
Conference Paper
Full-text available
Most storytelling games bring people together to co-create stories. However, they often require considerable creative effort and skills from all players, possibly discouraging less resourceful participants and impairing stories' quality. Moreover, most stories created within these games are usually only kept in players' minds rather than on storage, despite being a valuable and original asset, with a large potential for the narrative research community. We address these challenges with a novel mixed-initiative approach aimed at supporting a group of players to incrementally co-create a story, one sentence at a time. Our method features a hand generator that offers a unique set of tokens (words) to each player in each turn. This generator carefully combines tokens relevant to the ongoing story, to each individual player, to the group as a whole and random. We implemented this method in TaleMaker, a multiplayer online game that stimulates playful co-creation of a story. TaleMaker gives players considerable creative freedom to compose their sentences, combining a gentle structural steering with the wisdom of the group to determine the best direction for the story. The collected output of TaleMaker consists of annotated stories, with slots (e.g. action, character, location) filled with words associated with a Word-Net synset. From a preliminary evaluation, players reported that TaleMaker effectively stimulated story authoring, and perceived TaleMaker-created stories of considerable quality. In addition, a first analysis of the collected tokens confirms that players mostly collected story-related tokens, rather than those randomly offered. CCS CONCEPTS • Computing methodologies → Machine learning algorithms; • Human-centered computing → Collaborative and social computing theory, concepts and paradigms.
... In this study, we chose the GPT-2 model as an intermediate solution between speed and performance, since searching requires both parameters to be acceptable. Applications of text-generation AI for creative tasks often utilize machine-in-the-loop approach, where AI served only as a proposer (Osone et al., 2021;Roemmele & Gordon, 2015;Clark et al., 2018;Gero & Chilton, 2019). We kept the basic framework of the machine-in-the-loop approach, but further designed our tool in consideration of the design process features. ...
... Similarly, there are other tools meant for a specialized task or audience. For example, FLOW [10] is an interactive writing assistant for people who learn english as a foreign language, Creative Help [48] and LISA [49] help with story writing, SWAN [34] helps with scientific writing, etc. ...
... • It can prevent system failure due to an increase in the number and length of turns in dialogue systems [4]. • It is highly effective in systems such as extractive text summarization [5], story-telling text completion recommender [6], retrieval-based question answering (QA) [7] where the incorrect sequence of sentences may lead to a weak performance of the system and decline users' understanding. ...
Preprint
Full-text available
Building systems with capability of natural language understanding (NLU) has been one of the oldest areas of AI. An essential component of NLU is to detect logical succession of events contained in a text. The task of sentence ordering is proposed to learn succession of events with applications in AI tasks. The performance of previous works employing statistical methods is poor, while the neural networks-based approaches are in serious need of large corpora for model learning. In this paper, we propose a method for sentence ordering which does not need a training phase and consequently a large corpus for learning. To this end, we generate sentence embedding using BERT pre-trained model and measure sentence similarity using cosine similarity score. We suggest this score as an indicator of sequential events' level of coherence. We finally sort the sentences through brute-force search to maximize overall similarities of the sequenced sentences. Our proposed method outperformed other baselines on ROCStories, a corpus of 5-sentence human-made stories. The method is specifically more efficient than neural network-based methods when no huge corpus is available. Among other advantages of this method are its interpretability and needlessness to linguistic knowledge.
... Similarly, there are other tools meant for a specialized task or audience. For example, FLOW [10] is an interactive writing assistant for people who learn english as a foreign language, Creative Help [48] and LISA [49] help with story writing, SWAN [34] helps with scientific writing, etc. ...
Preprint
Full-text available
Stuttering is a speech disorder which impacts the personal and professional lives of millions of people worldwide. To save themselves from stigma and discrimination, people who stutter (PWS) may adopt different strategies to conceal their stuttering. One of the common strategies is word substitution where an individual avoids saying a word they might stutter on and use an alternative instead. This process itself can cause stress and add more burden. In this work, we present Fluent, an AI augmented writing tool which assists PWS in writing scripts which they can speak more fluently. Fluent embodies a novel active learning based method of identifying words an individual might struggle pronouncing. Such words are highlighted in the interface. On hovering over any such word, Fluent presents a set of alternative words which have similar meaning but are easier to speak. The user is free to accept or ignore these suggestions. Based on such user interaction (feedback), Fluent continuously evolves its classifier to better suit the personalized needs of each user. We evaluated our tool by measuring its ability to identify difficult words for 10 simulated users. We found that our tool can identify difficult words with a mean accuracy of over 80% in under 20 interactions and it keeps improving with more feedback. Our tool can be beneficial for certain important life situations like giving a talk, presentation, etc. The source code for this tool has been made publicly accessible at github.com/bhavyaghai/Fluent.
... • Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories first author, Elizabeth Clark: The significant distinctions are the writing flexibility and the intuitive options menu that enables adding images and texts. • Creative Help: A Story Writing Assistant [Roemmele and Gordon, 2015] first-author, Dr. Melissa Roemmele: The dominant strength is the analysis platform that can be used for evaluation thanks to the editable autocompletes and user-inputted content. The images are also novel in comparison to existing story generation platforms. ...
Preprint
Full-text available
Storytelling is an open-ended task that entails creative thinking and requires a constant flow of ideas. Natural language generation (NLG) for storytelling is especially challenging because it requires the generated text to follow an overall theme while remaining creative and diverse to engage the reader. In this work, we introduce a system and a web-based demo, FairyTailor, for human-in-the-loop visual story co-creation. Users can create a cohesive children's fairytale by weaving generated texts and retrieved images with their input. FairyTailor adds another modality and modifies the text generation process to produce a coherent and creative sequence of text and images. To our knowledge, this is the first dynamic tool for multimodal story generation that allows interactive co-formation of both texts and images. It allows users to give feedback on co-created stories and share their results.
... Much prior work focused on predicting the immediate future of a story, i.e., one to a few sentences later. For example, the Creative Help system used a recurrent neural network model to generate the next sentence to support writing (Roemmele and Gordon, 2015); the Scheherazade system uses crowdsourcing and artificial intelligence techniques to interactively construct the narrative sentence by sentence (Li and Riedl, 2015); Clark et al. (2018) study machine-in-the-loop story writing where the machine constantly generates a suggestion for the next sentence to stimulate writers; and Metaphoria (Gero and Chilton, 2019) generates metaphors, an even smaller unit, to inspire writers based on an input word by searching relations and ranking distances on ConceptNet (Liu and Singh, 2004). ...
... For story evaluation, human evaluators are asked to rate the generated stories based on different criteria such as consistency, coherence, and interestingness [108,129]. Another approach for story subjective evaluation is to ask evaluators to edit generated stories and calculate the story quality measure as the distance between the edit and the original story [90,144]. Regardless of the wide usage of human evaluation, it suffers from being inflexible, time/effort consuming, subjective, and it has no gold standard for comparing different story generation systems. ...
Article
Full-text available
Computational generation of stories is a subfield of computational creativity where artificial intelligence and psychology intersect to teach computers how to mimic humans’ creativity. It helps generate many stories with minimum effort and customize the stories for the users’ education and entertainment needs. Although the automatic generation of stories started to receive attention many decades ago, advances in this field to date are less than expected and suffer from many limitations. This survey presents an extensive study of research in the area of non-interactive textual story generation, as well as covering resources, corpora, and evaluation methods that have been used in those studies. It also shed light on factors of story interestingness.
... Much prior work focused on predicting the immediate future of a story, i.e., one to a few sentences later. For example, the Creative Help system used a recurrent neural network model to generate the next sentence to support writing (Roemmele and Gordon, 2015); the Scheherazade system uses crowdsourcing and artificial intelligence techniques to interactively construct the narrative sentence by sentence (Li and Riedl, 2015); Clark et al. (2018) study machine-in-the-loop story writing where the machine constantly generates a suggestion for the next sentence to stimulate writers; and Metaphoria (Gero and Chilton, 2019) generates metaphors, an even smaller unit, to inspire writers based on an input word by searching relations and ranking distances on ConceptNet (Liu and Singh, 2004). ...
Preprint
This paper introduces semantic frame forecast, a task that predicts the semantic frames that will occur in the next 10, 100, or even 1,000 sentences in a running story. Prior work focused on predicting the immediate future of a story, such as one to a few sentences ahead. However, when novelists write long stories, generating a few sentences is not enough to help them gain high-level insight to develop the follow-up story. In this paper, we formulate a long story as a sequence of "story blocks," where each block contains a fixed number of sentences (e.g., 10, 100, or 200). This formulation allows us to predict the follow-up story arc beyond the scope of a few sentences. We represent a story block using the term frequencies (TF) of semantic frames in it, normalized by each frame's inverse document frequency (IDF). We conduct semantic frame forecast experiments on 4,794 books from the Bookcorpus and 7,962 scientific abstracts from CODA-19, with block sizes ranging from 5 to 1,000 sentences. The results show that automated models can forecast the follow-up story blocks better than the random, prior, and replay baselines, indicating the task's feasibility. We also learn that the models using the frame representation as features outperform all the existing approaches when the block size is over 150 sentences. The human evaluation also shows that the proposed frame representation, when visualized as word clouds, is comprehensible, representative, and specific to humans. Our code is available at https://github.com/appleternity/FrameForecasting.
... • It can prevent system failure due to an increase in the number and length of turns in dialogue systems [4]. • It is highly effective in systems such as extractive text summarization [5], story-telling text completion recommender [6], retrieval-based question answering (QA) [7] where the incorrect sequence of sentences may lead to a weak performance of the system and decline users' understanding. ...
... Though successful in this regard, this success has only been realized in closed domains, where the narrative scaffolds can be specified in a limited ontology and the dynamics operations can be written by hand (such as e.g. the action schemata of Riedl and Young (2010a)). Neural generation has since helped scale to open domains (Roemmele and Gordon, 2015;Khalifa et al., 2017) but not with the same level of control over the narrative. Several recent works have looked at adding the narrative scaffolding component back into neural text generating systems Martin et al., 2018;Yao et al., 2019;Xu et al., 2018;Fan et al., 2019). ...
... The same is true of co-creative writing processes driven by language models [3,15,21,27]. Moreover, language model-based systems are particularly flawed from a creativity support perspective due to their lack of an explicit world model. ...
... Neural generation has since helped scale to open domains (Roemmele and Gordon, 2015;Khalifa et al., 2017) but not with the same level of control over the narrative. Several recent works have looked at adding the narrative scaffolding component back into neural text generating systems (Fan et al., 2018;Martin et al., 2018;Yao et al., 2019;Xu et al., 2018). ...
Preprint
Early work on narrative modeling used explicit plans and goals to generate stories, but the language generation itself was restricted and inflexible. Modern methods use language models for more robust generation, but often lack an explicit representation of the scaffolding and dynamics that guide a coherent narrative. This paper introduces a new model that integrates explicit narrative structure with neural language models, formalizing narrative modeling as a Switching Linear Dynamical System (SLDS). A SLDS is a dynamical system in which the latent dynamics of the system (i.e. how the state vector transforms over time) is controlled by top-level discrete switching variables. The switching variables represent narrative structure (e.g., sentiment or discourse states), while the latent state vector encodes information on the current state of the narrative. This probabilistic formulation allows us to control generation, and can be learned in a semi-supervised fashion using both labeled and unlabeled data. Additionally, we derive a Gibbs sampler for our model that can fill in arbitrary parts of the narrative, guided by the switching variables. Our filled-in (English language) narratives outperform several baselines on both automatic and human evaluations.
... Automatic story generation has a long history, with early work based primarily on hand-written rules (Klein et al., 1973;Meehan, 1977;Dehn, 1981;Turner, 1993). Subsequent methods were based on planning from artificial intelligence (Theune et al., 2003;Oinonen et al., 2006;Riedl and Young, 2010) and, more recently, data-driven methods have been developed (McIntyre and Lapata, 2010;Elson, 2012;Daza et al., 2016;Roemmele and Gordon, 2015;Clark et al., 2018a;Martin et al., 2018;Fan et al., 2018b;Yao et al., 2019;Fan et al., 2019). In concurrent work, Gupta et al. (2019) also propose methods to generate more diverse and interesting story endings, albeit without control variables. ...
... The perceived authenticity of such systems-be it in a conversational setting or not-will be key to these systems' success: in many domains, we can expect a more meaningful engagement with a system on the user's part, if the system features human-like behaviour. While we do not expect new rap songs penned by artificial ghostwriters any day soon, prior research has shown the value of collaborative NLG systems, in which humans are assisted by NLG systems in their writing process [21,44,45]. The experiments reported above show that for short textual fragments, restricted to a focused domain, encouraging results can be obtained for various model architectures. ...
Article
Full-text available
Through advances in neural language modeling, it has become possible to generate artificial texts in a variety of genres and styles. While the semantic coherence of such texts should not be over-estimated, the grammatical correctness and stylistic qualities of these artificial texts are at times remarkably convincing. In this paper, we report a study into crowd-sourced authenticity judgments for such artificially generated texts. As a case study, we have turned to rap lyrics, an established sub-genre of present-day popular music, known for its explicit content and unique rhythmical delivery of lyrics. The empirical basis of our study is an experiment carried out in the context of a large, mainstream contemporary music festival in the Netherlands. Apart from more generic factors, we model a diverse set of linguistic characteristics of the input that might have functioned as authenticity cues. It is shown that participants are only marginally capable of distinguishing between authentic and generated materials. By scrutinizing the linguistic features that influence the participants' authenticity judgments, it is shown that linguistic properties such as 'syntactic complexity', 'lexical diversity' and 'rhyme density' add to the user's perception of texts being authentic. This research contributes to the improvement of the quality and credibility of generated text. Additionally, it enhances our understanding of the perception of authentic and artificial art.
... Automatic story generation has a long history, with early work based primarily on hand-written rules (Klein et al., 1973;Meehan, 1977;Dehn, 1981;Turner, 1993). Subsequent methods were based on planning from artificial intelligence (Theune et al., 2003;Oinonen et al., 2006;Riedl and Young, 2010) and, more recently, data-driven methods have been developed (McIntyre and Lapata, 2010;Elson, 2012;Daza et al., 2016;Roemmele and Gordon, 2015;Clark et al., 2018a;Martin et al., 2018;Fan et al., 2018b;Yao et al., 2019;Fan et al., 2019). In concurrent work, Gupta et al. (2019) also propose methods to generate more diverse and interesting story endings, albeit without control variables. ...
Preprint
Full-text available
We propose a simple and effective modeling framework for controlled generation of multiple, diverse outputs. We focus on the setting of generating the next sentence of a story given its context. As controllable dimensions, we consider several sentence attributes, including sentiment, length, predicates, frames, and automatically-induced clusters. Our empirical results demonstrate: (1) our framework is accurate in terms of generating outputs that match the target control values; (2) our model yields increased maximum metric scores compared to standard n-best list generation via beam search; (3) controlling generation with semantic frames leads to a stronger combination of diversity and quality than other control variables as measured by automatic metrics. We also conduct a human evaluation to assess the utility of providing multiple suggestions for creative writing, demonstrating promising results for the potential of controllable, diverse generation in a collaborative writing system.
... Some prior work has explored interactive computersupported story writing, in which the system populates suggestions or inspirations for the writer in near real-time when he/she writes the story. For example, the Creative Help system generates suggestions for the next sentence in the process of story writing [12][13][14]. Clark et al. studied machinein-the-loop short story writing and concluded that machine intervention should balance between generating coherent and surprising suggestions [2]. Our study provides a detailed understanding of how users receive and edit machine-generated text in the context of story writing. ...
Conference Paper
A significant body of research in Artificial Intelligence (AI) has focused on generating stories automatically, either based on prior story plots or input images. However, literature has little to say about how users would receive and use these stories. Given the quality of stories generated by modern AI algorithms, users will nearly inevitably have to edit these stories before putting them to real use. In this paper, we present the first analysis of how human users edit machine-generated stories. We obtained 962 short stories generated by one of the state-of-the-art visual storytelling models. For each story, we recruited five crowd workers from Amazon Mechanical Turk to edit it. Our analysis of these edits shows that, on average, users (i) slightly shortened machine-generated stories, (ii) increased lexical diversity in these stories, and (iii) often replaced nouns and their determiners/articles with pronouns. Our study provides a better understanding on how users receive and edit machine-generated stories, informing future researchers to create more usable and helpful story generation systems.
... Some prior work has explored interactive computersupported story writing, in which the system populates suggestions or inspirations for the writer in near real-time when he/she writes the story. For example, the Creative Help system generates suggestions for the next sentence in the process of story writing [12][13][14]. Clark et al. studied machinein-the-loop short story writing and concluded that machine intervention should balance between generating coherent and surprising suggestions [2]. Our study provides a detailed understanding of how users receive and edit machine-generated text in the context of story writing. ...
Preprint
Full-text available
A significant body of research in Artificial Intelligence (AI) has focused on generating stories automatically, either based on prior story plots or input images. However, literature has little to say about how users would receive and use these stories. Given the quality of stories generated by modern AI algorithms, users will nearly inevitably have to edit these stories before putting them to real use. In this paper, we present the first analysis of how human users edit machine-generated stories. We obtained 962 short stories generated by one of the state-of-the-art visual storytelling models. For each story, we recruited five crowd workers from Amazon Mechanical Turk to edit it. Our analysis of these edits shows that, on average, users (i) slightly shortened machine-generated stories, (ii) increased lexical diversity in these stories, and (iii) often replaced nouns and their determiners/articles with pronouns. Our study provides a better understanding on how users receive and edit machine-generated stories,informing future researchers to create more usable and helpful story generation systems.
... Alternatively, a machine-in-the-loop paradigm places the human in the primary position of action and control while the machine assists. Examples of this include a creative-writing assistance system that generates contextual suggestions [8,38], and predicting situations in which people are likely to make errors in judgments in decision-making [1]. Even tasks which should not be automated may still benefit from machine assistance, especially if human performance is not the upper bound as Kleinberg et al. found in judge bail decisions [21]. ...
Preprint
Although artificial intelligence holds promise for addressing societal challenges, issues of exactly which tasks to automate and the extent to do so remain understudied. We approach the problem of task delegability from a human-centered perspective by developing a framework on human perception of task delegation to artificial intelligence. We consider four high-level factors that can contribute to a delegation decision: motivation, difficulty, risk, and trust. To obtain an empirical understanding of human preferences in different tasks, we build a dataset of 100 tasks from academic papers, popular media portrayal of AI, and everyday life. For each task, we administer a survey to collect judgments of each factor and ask subjects to pick the extent to which they prefer AI involvement. We find little preference for full AI control and a strong preference for machine-in-the-loop designs, in which humans play the leading role. Our framework can effectively predict human preferences in degrees of AI assistance. Among the four factors, trust is the most predictive of human preferences of optimal human-machine delegation. This framework represents a first step towards characterizing human preferences of automation across tasks. We hope this work may encourage and aid in future efforts towards understanding such individual attitudes; our goal is to inform the public and the AI research community rather than dictating any direction in technology development.
Article
While developing a story, novices and published writers alike have had to look outside themselves for inspiration. Language models have recently been able to generate text fluently, producing new stochastic narratives upon request. However, effectively integrating such capabilities with human cognitive faculties and creative processes remains challenging. We propose to investigate this integration with a multimodal writing support interface that offers writing suggestions textually, visually, and aurally. We conduct an extensive study that combines elicitation of prior expectations before writing, observation and semi-structured interviews during writing, and outcome evaluations after writing. Our results illustrate individual and situational variation in machine-in-the-loop writing approaches, suggestion acceptance, and ways the system is helpful. Centrally, we report how participants perform integrative leaps , by which they do cognitive work to integrate suggestions of varying semantic relevance into their developing stories. We interpret these findings, offering modeling and design recommendations for future creative writing support technologies.
Article
Full-text available
Interest in artificial intelligence (AI) language models has grown considerably following the release of ‘generative pre-trained transformer’ (GPT). Framing AI as an extractive technology, this article details how GPT harnesses human labour and sensemaking at two stages: (1) during training when the algorithm ‘learns’ biased communicative patterns extracted from the Internet and (2) during usage when humans write alongside the AI. This second phase is framed critically as a form of unequal ‘affective labour’ where the AI imposes narrow and biased conditions for the interaction to unfold, and then exploits the resulting affective turbulence to sustain its simulation of autonomous performance. Empirically, this article draws on an in-depth case study where a human engaged with an AI writing tool, while the researchers recorded the interactions and collected qualitative data about perceptions, frictions and emotions.
Chapter
The increase in development of machines that have capability to understand the complicated behavior to solve the human brain involvement problems, the auto text generation application also gets the wide attention. The Language modeling or text generation is a task of next character or word prediction in a sequence with analysis of input data. The ATG enable the machines to write and provide the help to reduce the human brain effort. The ATG is also useful for understanding and analysis of languages and provide the techniques that enable the machines to exchange information in natural languages. At the large scale the text data are created everywhere (whatsApp, facebook, and tweets etc.) and freely online available therefore an effective system is needed for automation of text generation process and analysis of the text data for extracting meaningful information from it so in this work, a case study is presented on how develop a text generation model using hybrid recurrent neural network for English language. The explore model find the dependencies between characters and the conditional probabilities of character in sequences from the available input text data and generate the wholly new sequences of characters like human beings writing (correct in meaning, spelling and sentence structure). A comprehensive comparison between these models, namely, LSTM, deep LSTM, GRU and HRNN is also presented. Previously the RNN models are used for text predictions or auto text generation but these models created the problem of vanishing gradient (short memory) when process long text, therefore the GRU and LSTM models were created for solving this problem. The text generated by GRU and LSTM have many spellings error, incorrect sentence structure, therefore, filling this gap the HRNN model is explore. The HRNN model is the combination of LSTM, GRU and a dense layer. The experiments performed on Penn Treebank, Shakespeare, and Nietzsche datasets. The perplexity of HRNN model is 3.27, the bit per character is 1.18 and average word prediction accuracy is 0.63. As compare with baseline work and previous models (LSTM, deep LSTM and GRU), our model (HRNN) perplexity and bit per character are less. The texts generated by HRNN have fewer spelling errors and sentence structure mistakes. A closer analysis of explored models’ performance and efficiency is described with the help of graph plots and generated texts by taking some input strings. These graphs explain the performance for each model.
Conference Paper
Full-text available
This paper proposes a generative language model called AfriKI. Our approach is based on an LSTM architecture trained on a small corpus of contemporary fiction. With the aim of promoting human creativity, we use the model as an authoring tool to explore machine-in-the-loop Afrikaans poetry generation. To our knowledge, this is the first study to attempt creative text generation in Afrikaans.
Preprint
Full-text available
This paper proposes a generative language model called AfriKI. Our approach is based on an LSTM architecture trained on a small corpus of contemporary fiction. With the aim of promoting human creativity, we use the model as an authoring tool to explore machine-in-the-loop Afrikaans poetry generation. To our knowledge, this is the first study to attempt creative text generation in Afrikaans.
Article
The computational narrative generation studies have got attention in various genres and media from novels to movie scripts. Recently, robots are replacing humans in writing informative and straightforward texts. However, generating narratives is still challenging since it requires high-level perceptions, including imagination and creativity. This study aims to review technological advances in the computational narrative generation. We first introduce narratology theories that are foundations of this research area. Then, we show the existing narrative generation tools, which are based on narratology theories. Also, there are generation tools based on deep learning techniques. We compare them with conventional methods and present their limitations. Finally, we suggest further research directions in the computational narrative generation.
Conference Paper
Full-text available
We describe the design and use of the Stanford CoreNLP toolkit, an extensible pipeline that provides core natural language analysis. This toolkit is quite widely used, both in the research NLP community and also among commercial and government users of open source NLP technology. We suggest that this follows from a simple, approachable design, straight-forward interfaces, the inclusion of robust and good quality analysis components, and not requiring use of a large amount of associated baggage.
Conference Paper
Full-text available
Personal narratives typically involve a narrator who participates in a sequence of events in the past. The narrator is therefore present at two narrative levels: (1) the extradiegetic level, where the act of narration takes place, with the narrator addressing an audience directly; and (2) the diegetic level, where the events in the story take place, with the narrator as a participant (usually the protagonist). Although story understanding is commonly associated with semantics of the diegetic level (i.e., understanding the events that take place within the story), personal narratives may also contain important information at the extradiegetic level that frames the narrated events and is crucial for capturing the narrator's intent. We present a data-driven modeling approach that learns to identify subjective passages that express mental and emotional states of the narrator, placing them at either the diegetic or extradiegetic level. We describe an experiment where we used narratives from personal weblog posts to measure the effectiveness of our approach across various topics in this narrative genre.
Conference Paper
Full-text available
If we take an existing supervised NLP sys- tem, a simple and general way to improve accuracy is to use unsupervised word representations as extra word features. We evaluate Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeddings of words on both NER and chunking. We use near state-of-the-art supervised baselines, and find that each of the three word representations improves the accu- racy of these baselines. We find further improvements by combining di erent word representations. You can download our word features, for o -the-shelf use in existing NLP systems, as well as our code, here: http://metaoptimize. com/projects/wordreprs/
Article
Full-text available
MEXICA is a computer model that produces frameworks for short stories based on the engagement-reflection cognitive account of writing. During engagement MEXICA generates material guided by content and rhetorical constraints, avoiding the use of explicit goals or story-structure information. During reflection the system breaks impasses, evaluates the novelty and interestingness of the story in progress and verifies that coherence requirements are satisfied. In this way, MEXICA complements and extends those models of computerised story-telling based on traditional problem-solving techniques where explicit goals drive the generation of stories. This paper describes the engagement-reflection account of writing, the general characteristics of MEXICA and reports an evaluation of the program.
Article
Story generation is the problem of automatically selecting a sequence of events that meet a set of criteria and can be told as a story. Story generation is knowledge-intensive; traditional story generators rely on a priori defined domain models about fictional worlds, including characters, places, and actions that can be performed. Manually authoring the domain models is costly and thus not scalable. We present a novel class of story generation system that can generate stories in an unknown domain. Our system (a) automatically learns a domain model by crowdsourcing a corpus of narrative examples and (b) generates stories by sampling from the space defined by the domain model. A large-scale evaluation shows that stories generated by our system for a previously unknown topic are comparable in quality to simple stories authored by untrained humans. © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
Article
We describe Say Anything, a new interactive storytelling system that collaboratively writes textual narratives with human users. Unlike previous attempts, this interactive storytelling system places no restrictions on the content or direction of the user’s contribution to the emerging storyline. In response to these contributions, the computer continues the storyline with narration that is both coherent and entertaining. This capacity for open-domain interactive storytelling is enabled by an extremely large repository of nonfiction personal stories, which is used as a knowledge base in a case-based reasoning architecture. In this article, we describe the three main components of our case-based reasoning approach: a million-item corpus of personal stories mined from internet weblogs, a case retrieval strategy that is optimized for narrative coherence, and an adaptation strategy that ensures that repurposed sentences from the case base are appropriate for the user’s emerging fiction. We describe a series of evaluations of the system’s ability to produce coherent and entertaining stories, and we compare these narratives with single-author stories posted to internet weblogs.
Article
We propose a new deterministic approach to coreference resolution that combines the global information and precise features of modern machine-learning models with the transparency and modularity of deterministic, rule-based systems. Our sieve architecture applies a battery of deterministic coreference models one at a time from highest to lowest precision, where each model builds on the previous model's cluster output. The two stages of our sieve-based architecture, a mention detection stage that heavily favors recall, followed by coreference sieves that are precision-oriented, offer a powerful way to achieve both high precision and high recall. Further, our approach makes use of global information through an entity-centric model that encourages the sharing of features across all mentions that point to the same real-world entity. Despite its simplicity, our approach gives state-of-the-art performance on several corpora and genres, and has also been incorporated into hybrid state-of-the-art coreference systems for Chinese and Arabic. Our system thus offers a new paradigm for combining knowledge in rule-based systems that has implications throughout computational linguistics.
Article
Stories of people's everyday experiences have long been the focus of psychology and sociology research, and are in-creasingly being used in innovative knowledge-based tech-nologies. However, continued research in this area is hin-dered by the lack of standard corpora of sufficient size and by the costs of creating one from scratch. In this paper, we describe our efforts to develop a standard corpus for re-searchers in this area by identifying personal stories in the tens of millions of blog posts in the ICWSM 2009 Spinn3r Dataset. Our approach was to employ statistical text classi-fication technology on the content of blog entries, which re-quired the creation of a sufficiently large set of annotated training examples. We describe the development and evaluation of this classification technology and how it was applied to the dataset in order to identify nearly a million personal stories.
Article
The generation of extended plots for melodramatic fiction is an interesting task for Artificial Intelligence research, one that requires the application of generalization techniques to carry out fully. UNIVERSE is a story-telling program that uses plan-like units, ‘plot fragments’, to generate plot outlines. By using a rich library of plot fragments and a well-developed set of characters, UNIVERSE can create a wide range of plot outlines. In this paper we illustrate how UNIVERSE's plot fragment library is used to create plot outlines and how it might be automatically extended using explanation-based generalization methods. Our methods are based on analysis of a television melodrama, including comparisons of similar stories.
Conference Paper
In this paper we describe the architecture of an interactive story generation system where a human and computer each take turns writing sentences of an emerging narrative. Each turn begins with the user adding a sentence to the story, where the computer responds with a sentence of its own that continues what has been written so far. Rather than generating the next sentence from scratch, the computer selects the next sentence from a corpus of tens of millions of narrative sentences extracted from Internet weblogs. We compare five different retrieval methods for selecting the most appropriate sentence, and present the results of a user study to determine which of these models produces stories with the highest coherence and overall value.
Conference Paper
Hand-coded scripts were used in the 1970-80s as knowledge backbones that enabled infer- ence and other NLP tasks requiring deep se- mantic knowledge. We propose unsupervised induction of similar schemata called narrative event chains from raw newswire text. A narrative event chain is a partially ordered set of events related by a common protago- nist. We describe a three step process to learn- ing narrative event chains. The first uses unsu- pervised distributional methods to learn narra- tive relations between events sharing corefer- ring arguments. The second applies a tempo- ral classifier to partially order the connected events. Finally, the third prunes and clusters self-contained chains from the space of events. We introduce two evaluations: the narrative cloze to evaluate event relatedness, and an or- der coherence task to evaluate narrative order. We show a 36% improvement over baseline for narrative prediction and 25% for temporal coherence.
Article
'Measuring dialect pronunciation differences using Levenshtein distance' is het proefschrift van Wilbert Heeringa. Hij heeft zich beziggehouden met een onderzoek naar methoden waarmee taalvariatie nader bestudeerd kan worden. Heeringa bespreekt eerst diverse methoden voor het meten van taalkundige afstanden tussen dialecten, waaronder de 'Levenshtein-afstand'. Deze methode staat centraal in het verdere proefschrift; er worden verschillende varianten van gepresenteerd. Vervolgens legt Heeringa uit hoe de betreffende methode precies werkt en meet. Dan volgt de beschrijving van een aantal onderzoeken waarin de methode wordt gebruikt. Een groot aantal Nederlandse dialecten wordt hierin met elkaar en met het standaard Nederlands vergeleken. Ook wordt een aantal Noorse dialecten bij het onderzoek betrokken. Het boek besluit met de conclusie, die luidt dat de Levenshtein-afstand bruikbaar is voor het berekenen van afstanden tussen taalvariëteiten. Dit is gebaseerd op de uitkomsten van gepresenteerde onderzoeken. Bron: http://taalunieversum.org/cultuur/bibliografische attenderingslijst/20/3/2/
Minimal narrative annotation schemes and their applications
  • E Rahimtoroghi
  • T Corcoran
  • R Swanson
  • M A Walker
  • K Sagae
  • A S Gordon