ArticlePublisher preview available

On user rationale in software engineering

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

Rationale refers to the reasoning and justification behind human decisions, opinions, and beliefs. In software engineering, rationale management focuses on capturing design and requirements decisions and on organizing and reusing project knowledge. This paper takes a different view on rationale written by users in online reviews. We studied 32,414 reviews for 52 software applications in the Amazon Store. Through a grounded theory approach and peer content analysis, we investigated how users argue and justify their decisions, e.g., about upgrading, installing, or switching software applications. We also studied the occurrence frequency of rationale concepts such as issues encountered or alternatives considered in the reviews and found that assessment criteria like performance, compatibility, and usability represent the most pervasive concept. We identified a moderate positive correlation between issues and criteria and furthermore assessed the distribution of rationale concepts with respect to rating and verbosity. We found that issues tend to appear more in lower star rated reviews, while criteria, alternatives, and justifications seem to appear more in three star rated reviews. Also, reviews reporting alternatives seem to be more verbose than reviews reporting criteria. A follow-up qualitative study of sub-concepts revealed, that users also report other alternatives (e.g., alternative software provider), criteria (e.g., cost), and decisions (e.g., on rating software). We then used the truth set of manually labeled review sentences to explore how accurately we can mine rationale concepts from the reviews. We evaluated the classification algorithms Naive Bayes, Support Vector Machine, Logistic Regression, Decision Tree, Gaussian Process, Random Forest, and Multilayer Perceptron Classifier using a baseline and random configuration. Support Vector Classifier, Naive Bayes, and Logistic Regression, trained on the review metadata, syntax tree of the review text, and influential terms, achieved a precision around 80% for predicting sentences with alternatives and decisions, with top recall values of 98%. On the review level, precision was up to 13% higher with recall values reaching 99%. Using only word features, we achieved in most cases the highest precision and highest recall respectively using the Random Forest and Naive Bayes algorithm. We discuss the findings and the rationale importance for supporting deliberation in user communities and synthesizing the reviews for developers.
This content is subject to copyright. Terms and conditions apply.
Vol.:(0123456789)
1 3
Requirements Engineering (2018) 23:357–379
https://doi.org/10.1007/s00766-018-0293-2
RE 2017
On user rationale insoftware engineering
ZijadKurtanović1 · WalidMaalej1
Received: 21 October 2017 / Accepted: 23 March 2018 / Published online: 6 April 2018
© Springer-Verlag London Ltd., part of Springer Nature 2018
Abstract
Rationale refers to the reasoning and justification behind human decisions, opinions, and beliefs. In software engineering,
rationale management focuses on capturing design and requirements decisions and on organizing and reusing project knowl-
edge. This paper takes a different view on rationale written by users in online reviews. We studied 32,414 reviews for 52
software applications in the Amazon Store. Through a grounded theory approach and peer content analysis, we investigated
how users argue and justify their decisions, e.g., about upgrading, installing, or switching software applications. We also
studied the occurrence frequency of rationale concepts such as issues encountered or alternatives considered in the reviews
and found that assessment criteria like performance, compatibility, and usability represent the most pervasive concept. We
identified a moderate positive correlation between issues and criteria and furthermore assessed the distribution of rationale
concepts with respect to rating and verbosity. We found that issues tend to appear more in lower star rated reviews, while
criteria, alternatives, and justifications seem to appear more in three star rated reviews. Also, reviews reporting alternatives
seem to be more verbose than reviews reporting criteria. A follow-up qualitative study of sub-concepts revealed, that users
also report other alternatives (e.g., alternative software provider), criteria (e.g., cost), and decisions (e.g., on rating software).
We then used the truth set of manually labeled review sentences to explore how accurately we can mine rationale concepts
from the reviews. We evaluated the classification algorithms Naive Bayes, Support Vector Machine, Logistic Regression,
Decision Tree, Gaussian Process, Random Forest, and Multilayer Perceptron Classifier using a baseline and random con-
figuration. Support Vector Classifier, Naive Bayes, and Logistic Regression, trained on the review metadata, syntax tree of
the review text, and influential terms, achieved a precision around 80% for predicting sentences with alternatives and deci-
sions, with top recall values of 98%. On the review level, precision was up to 13% higher with recall values reaching 99%.
Using only word features, we achieved in most cases the highest precision and highest recall respectively using the Random
Forest and Naive Bayes algorithm. We discuss the findings and the rationale importance for supporting deliberation in user
communities and synthesizing the reviews for developers.
Keywords App analytics· Rationale· Review mining
1 Introduction
According to Merriam-Webster [40], rationale is “the expla-
nation of controlling principles of opinion, belief, practice,
or phenomena, or an underlying reason.” Rationale man-
agement focuses on capturing and sharing the reasons and
justifications behind decisions. The design rationale soft-
ware design paradigm aims at capturing design decisions
and their justifications [4]. Over the last decades, managing
requirements and design rationale has been a major concern
in software engineering [5, 12, 23]. Ideally, rationale should
be captured in requirements and design artifacts to docu-
ment why certain project decisions were taken [13]. This
includes the questions or issues encountered by designers
and analysts, the alternatives explored to solve the issues,
and the criteria to evaluate the alternatives [4]. Rationale is
also often found in informal artifacts such as team conversa-
tions or sketches [28].
With the increasing popularity of social media, user
forums, and app stores, software vendors started giving
more attention to the input of users when making deci-
sions about software design, development, and evolution
* Zijad Kurtanović
kurtanovic@informatik.uni-hamburg.de
Walid Maalej
maalej@informatik.uni-hamburg.de
1 University ofHamburg, Hamburg, Germany
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
... Intent identi cation involves determining customer intent based on online user reviews [7]. Also, previous studies demonstrate that crowd comments from end-users mobile stores [8], Tweets [9], and posts [10,11] include important and well-informed information regarding the user's interaction [12], requests for new features [13,14], and enduser rationale information [15,16], which is used to make informed requirements decision-making [17,18]. ...
... Finally, the least prominent rationale element in the data set is the decision, which is 4% (659) end-user comments of an overall 11416 end-user comments annotation sample. With the decision rationale element, software developers and requirements engineers can take pivotal requirements-related decisions to improve the performance of the existing software applications, i-e, why certain crowd-users ended up leaving the software application [15,18]. Additionally, while analyzing the end-user comments in the Amazon store, we found that crowd-user register negative comments against the software application, while its corresponding rating score is either 5 or 4, representing supporting-claim. ...
... For this ML experiment, we employed text preprocessing and feature engineering techniques to assess and analyze the various classi ers and reveal their performance in automatically identifying rationale elements in the end-user reviews. Hence, before text preprocessing, feature engineering, data balancing, and training, we rst selected different machine learning algorithms based on their good performance on textual data mentioned in the literature [15,17,35]. The selected algorithms are MNB, SVM, LR, KNN, MLP, GB, Voting classi er, RF, and Ensemble Methods. ...
Preprint
Full-text available
Rationale refers to making human judgments, sets of reasons, or intentions to explain a particular decision. Nowadays, crowd-users argue and justify their decisions on social media platforms about market-driven software applications, thus generating a software rationale. Such rationale information can be of pivotal importance for the software and requirements engineers to enhance the performance of existing software applications by revealing end-users tactic knowledge to improve software designing and development decision-making. For this purpose, we proposed an automated approach to capture and analyze end-user reviews containing rationale information, focusing on low-rating applications in the amazon store using Natural Language Processing (NLP) and supervised machine learning (ML) classification methods. In the literature, high-rating applications have been emphasized while ignoring low-rating software application that causes potential biasness. Therefore, we examined 59 comparatively low-ranked market-based software applications from the Amazon app store covering various software categories to capture and identify crowd-users justifications. Next, using a developed grounded theory and content analysis approach, we studied and recorded how crowd-users analyze and explain their rationale based on issues encountered, attacking or supporting arguments registered, and updating or uninstalling software applications. Also, to achieve the best results, an experimental study is conducted by comparing various ML algorithms, i.e., MNB, LR, RF, MLP, KNN, AdaBoost, and Voting classifier, on the end-users rationale data set by preprocessing the input data, applying feature engineering, balancing the data set, and then training and testing the ML algorithms with a standard cross-validation approach. We obtained satisfactory results with MLP, voting, and RF Classifiers, having 93%, 93%, and 90% average accuracy, respectively. Also, we plot the ROC curves for the high-performing ML Classifier to identify and capture classifiers yielding the best performance with an under-sampling or oversampling balancing approach. Additionally, we obtained the average Precision, Recall, and F-measure values of 98%, 94%, 96%, 97%, 95%, and 96% for identifying supporting & decision rationale elements in the user comments, respectively. The proposed research approach outer-perform the existing rationale approaches with better Precision, Recall, and F-measure values.
... 11 The relevant social media platforms and user forums (Reddit) provide another channel to interact with diverse users and collect rich rationale information. 9,18 Researchers have recently set out to mine intuitive requirements and design rationale information from the user forums, 3,9 issue tracking system, 10,11 Twitter, 19 and user reviews. 20 Furthermore, researchers have recently utilized argumentation theory in the RE domain, which is widely accepted and supported by the RE research community. ...
... 11 The relevant social media platforms and user forums (Reddit) provide another channel to interact with diverse users and collect rich rationale information. 9,18 Researchers have recently set out to mine intuitive requirements and design rationale information from the user forums, 3,9 issue tracking system, 10,11 Twitter, 19 and user reviews. 20 Furthermore, researchers have recently utilized argumentation theory in the RE domain, which is widely accepted and supported by the RE research community. ...
... More recently, software researchers and software tool vendors have started developing different approaches to filter, analyze, validate automatically, and synthesize software artifacts, developer's chat, and user feedback into actionable decisions or suggestions for the software developers. The main focus of these works were to identify useful and instinctive information for software developers, designers, and requirements engineers present in the large amount of user feedback's and software artifacts from different sources, such as: Internet relay chat messages, 10 issue tracking system, 44 bug reports and design session documents, 11 Amazon store, 9 and design artifact documents. 14, 45 Alkadhi et al. 10 applied different classifiers to automatically capture rationale information in the Internet relay chat messages and classify them into fine-grained rationale elements, that is, alternative, issues, cons & pro arguments, and decisions. ...
Article
Full-text available
User forums enable a large population of crowd‐users to publicly share their experience, useful thoughts, and concerns about the software applications in the form of user reviews. Recent research studies have revealed that end‐user reviews contain rich and pivotal sources of information for the software vendors and developers that can help undertake software evolution and maintenance tasks. However, such user‐generated information is often fragmented, with multiple viewpoints from various stakeholders involved in the ongoing discussions in the Reddit forum. In this article, we proposed a crowd‐based requirements engineering by valuation argumentation (CrowdRE‐VArg) approach that analyzes the end‐users discussion in the Reddit forum and identifies conflict‐free new features, design alternatives, or issues, and reach a rationale‐based requirements decision by gradually valuating the relative strength of their supporting and attacking arguments. The proposed approach helps to negotiate the conflict over the new features or issues between the different crowd‐users on the run by finding a settlement that satisfies the involved crowd‐users in the ongoing discussion in the Reddit forum using argumentation theory. For this purpose, we adopted the bipolar gradual valuation argumentation framework, extended from the abstract argumentation framework and abstract valuation framework. The automated CrowdRE‐VArg approach is illustrated through a sample crowd‐users conversation topic adopted from the Reddit forum about Google Map mobile application. Finally, we applied natural language processing and different machine learning algorithms to support the automated execution of the CrowdRE‐VArg approach. The results demonstrate that the proposed CrowdRE‐VArg approach works as a proof‐of‐concept and automatically identifies prioritized requirements‐related information for software engineers.
... Similarly, app store feedback can be classified by their reported requirements type [90,92]; this could help distinguish reviews reporting functional requirements from those reporting non-functional requirements [68,93,94]; distilling non-functional requirements into fine-grained quality categories such as reliability, performance, or efficiency [95,96]. Another key use of the classification task is rationale mining; it involves detecting types of argumentations and justification users describe in reviews when making certain decisions, e.g. about upgrading, installing, or switching apps [97,98,99]. ...
... Scholars have shown an interest in manual analysis of app reviews [80,98]. The technique is used to facilitate Content Analysis e.g., to understand topics users discuss [2,11,72] and to develop a ground truth dataset for training and evaluating mining techniques [4,97]. ...
... text length, tense of text [97,98], importance of words e.g., td-idf [72,95], a word sequence e.g., n-gram [12,26] as well as linguistic analysis e.g., dependency relationship [176]. These properties are commonly combined with non-textual properties like user sentiment [15,177], review rating [97] or app category [158]. ...
Thesis
Full-text available
The thesis studies how mining app reviews can support software engineering. App reviews —short user reviews of an app in app stores— provide a potentially rich source of information to help software development teams maintain and evolve their products. Exploiting this information is however difficult due to the large number of reviews and the difficulty in extracting useful actionable information from short informal texts. A variety of app review mining techniques have been proposed to classify reviews and to extract information such as feature requests, bug descriptions, and user sentiments but the usefulness of these techniques in practice is still unknown. Research in this area has grown rapidly, resulting in a large number of scientific publications (at least 182 between 2010 and 2020) but nearly no independent evaluation and description of how diverse techniques fit together to support specific software engineering tasks have been performed so far. The thesis presents a series of contributions to address these limitations. We first report the findings of a systematic literature review in app review mining exposing the breadth and limitations of research in this area. Using findings from the literature review, we then present a reference model that relates features of app review mining tools to specific software engineering tasks supporting requirements engineering, software maintenance and evolution. We then present two additional contributions extending previous evaluations of app review mining techniques. We present a novel independent evaluation of opinion mining techniques using an annotated dataset created for our experiment. Our evaluation finds lower effectiveness than initially reported by the techniques authors. A final part of the thesis, evaluates approaches in searching for app reviews pertinent to a particular feature. The findings show a general purpose search technique is more effective than the state-of-the-art purpose-built app review mining techniques; and suggest their usefulness for requirements elicitation. Overall, the thesis contributes to improving the empirical evaluation of app review mining techniques and their application in software engineering practice. Researchers and developers of future app mining tools will benefit from the novel reference model, detailed experiments designs, and publicly available datasets presented in the thesis.
... Reviews discussing NFRs can serve as ad-hoc documentations of user requirements [44], [56], [57]. Such specification may be later used by software engineer to justify reasons why certain decisions has been made e.g., why certain mobile phone model has been supported, or why certain security protection mechanism has been implemented [51]. • Requirements Prioritization, Requested Modification Prioritization: Added with statistics, reviews reporting NFRs may help development team to prioritize their works [13], [52], [57]. ...
... • Requirements Prioritization, Requested Modification Prioritization: Added with statistics, reviews reporting NFRs may help development team to prioritize their works [13], [52], [57]. Comparing how frequently NFRs are reported, for how long do user reported them and whether these numbers are increasing or decreasing may indicate users relative importance of these requirements [51], [73]. ...
Conference Paper
Full-text available
App reviews can provide valuable information about user needs but analyzing them manually is challenging due to their large quantity and noisy nature. To overcome this problem, a variety of app review mining techniques have been proposed. So far, however, research in this area has paid little attention to the software engineering use cases of the mining techniques. This limits the understanding of their usefulness, applications and desired future developments. We address this problem by elabo- rating a reference model relating app review mining techniques to specific software engineering activities. In this paper, we present a unified description of software engineering use cases for mining app reviews and define a reference architecture realizing these use cases through a combination of natural language processing and data mining techniques. The use cases provide a novel systematic exposition of the envisioned applications and benefits of app review mining for software engineers. The reference architecture synthesises the diversity of research to realise these benefits and provide a general framework guiding the development and evaluation of future research and tools.
... These classifications are from different points of view, e.g., whether or not the reviews include bug information, requests for new features [40], whether they are informative [6], whether reviews across different languages and platforms are similar [46], or based on a taxonomy relevant to software maintenance and evolution [48], etc. Other studies focused on the information extraction from app reviews considering the fact that reading through the entire reviews is impractical [12,15,16,30,33,57]. For example, the types of complains [30], the app aspects loved by users [15], user rationale [33] and summaries for guiding release planning [56] are extracted and summarized for facilitating the review understanding. ...
... Other studies focused on the information extraction from app reviews considering the fact that reading through the entire reviews is impractical [12,15,16,30,33,57]. For example, the types of complains [30], the app aspects loved by users [15], user rationale [33] and summaries for guiding release planning [56] are extracted and summarized for facilitating the review understanding. ...
Preprint
User reviews of mobile apps provide a communication channel for developers to perceive user satisfaction. Many app features that users have problems with are usually expressed by key phrases such as "upload pictures", which could be buried in the review texts. The lack of fine-grained view about problematic features could obscure the developers' understanding of where the app is frustrating users, and postpone the improvement of the apps. Existing pattern-based approaches to extract target phrases suffer from low accuracy due to insufficient semantic understanding of the reviews, thus can only summarize the high-level topics/aspects of the reviews. This paper proposes a semantic-aware, fine-grained app review analysis approach (SIRA) to extract, cluster, and visualize the problematic features of apps. The main component of SIRA is a novel BERT+Attr-CRF model for fine-grained problematic feature extraction, which combines textual descriptions and review attributes to better model the semantics of reviews and boost the performance of the traditional BERT-CRF model. SIRA also clusters the extracted phrases based on their semantic relations and presents a visualization of the summaries. Our evaluation on 3,426 reviews from six apps confirms the effectiveness of SIRA in problematic feature extraction and clustering. We further conduct an empirical study with SIRA on 318,534 reviews of 18 popular apps to explore its potential application and examine its usefulness in real-world practice.
... Rationale management foresees how to capture and record the justification for why a certain decision was taken 9 . A series of software decisions are taken when designing and developing a software application 10 . ...
... The use of content analysis in the software engineering context is not new to software engineering research. Recent studies, such as [7] and [13], advocate a rigorous process for its use in software engineering. ...
Preprint
Full-text available
Analytics plays a crucial role in the data-informed decision-making processes of modern businesses. Unlike established software companies, software startups are not seen utilizing the potential of analytics even though a startup process should be primarily data-driven. There has been little understanding in the literature about analytics for software startups. This study set out to address the knowledge gap by exploring how analytics is understood in the context of software startups. To this end, we collected the qualitative data of three analytics platforms that are mostly used by startups from multiple sources. We covered platform documentation as well as experience reports of the software startups using these platforms. The data was analyzed using content analysis techniques. Four high-level concepts were identified that encapsulate the real understanding of software startups on analytics, including instrumentation of analytics, experimentation, diagnostic analysis, and getting insights. The first concept describes how startups set up analytics and the latter three illustrate the usage scenarios of analytics. This study is the first step toward understanding analytics in the software startup context. The identified concepts can guide further investigation of analytics in this context. It also provides some insights for software startups to set up analytics for data-informed decisions. Given the limitation of the data used in the study, the immediate next step is to ground as well as validate the acquired understanding using the primary data, by directly interacting with software startups.
... Murukannaiah et al. [37] proposed an Arg-ACH approach to resolve stakeholder's goals conflicts by identifying and capturing inconsistencies between the goals and beliefs. Kurtanovic and Maalej [38,39] applied supervised machine learning techniques on amazon store to mine rationale concepts from user reviews. Reviews are classified into different rationale categories: issues, alternatives, criteria, decision, and justification, and also found that how users argue and support their decision regarding up-gradation and application switching. ...
Article
Full-text available
In social media platforms, crowd-users extensively interact and contribute information related to software applications. Usually, crowd-users discuss software features or hot issues and record their opinions about the software applications under discussion either in textual form or via end-user votes. Such requirements-related information is considered a pivotal alternative source for requirements engineers to the already existing in-house stakeholders in order to illustrate decision-making. Also, requirements decision-making for Crowd requirements engineering is a difficult task, as it is always based on incomplete knowledge and requires trade-offs from multi-perspectives. However, existing requirements models and associated tools are still lacking, which enable requirements engineers to make informed decision-making and capture conflicting requirements knowledge. This paper elaborates the interaction among the crowd-users about the Google Map mobile application in the Reddit forum to recover conflicting requirements-related information using the goal modeling approach. For this purpose, we extracted critical arguments from a crowd-users conversation in user forums regarding a given design; built a graphical argumentation model based on the extracted information; aligned types of arguments with goal-oriented modeling constructs in the non-functional requirements framework; conducted exiting goal-model analysis to the requirements model to reach consensus based on argumentation and reasoning, such as supporting, attacking, undefined, and conflicting. The proposal is described with illustrative example models and the associated evaluation processes of design decision-making situation for Google Map interface design.
Article
Full-text available
We review the scholarly contributions that utilise natural language processing (NLP) techniques to support the design process. Using a heuristic approach, we gathered 223 articles that are published in 32 journals within the period 1991–present. We present state-of-the-art NLP in-and-for design research by reviewing these articles according to the type of natural language text sources: internal reports, design concepts, discourse transcripts, technical publications, consumer opinions and others. Upon summarising and identifying the gaps in these contributions, we utilise an existing design innovation framework to identify the applications that are currently being supported by NLP. We then propose a few methodological and theoretical directions for future NLP in-and-for design research.
Article
Full-text available
Newsrooms are still searching for ways to manage user comments because of both a desire for professional distance from their audiences and a lack of analytical tools. This paper presents findings from our exploratory, interdisciplinary study in journalism research and computer science that focuses on the algorithmic classification and clustering of user comments. In contrast to endeavours that aim at filtering out hate speech or spam, we take a more constructive approach and focus on detecting particularly useful or high-quality user contributions that can be leveraged for journalistic purposes. On the basis of a literature review and our own preliminary research on audience participation and user review analytics, we developed a mock-up of a software framework to help journalists systematically analyze user comments to this end. We then surveyed its effectiveness through two group discussions - one with comment moderators and another with editors from different editorial departments of a large German online newsroom. Features that journalists and comment moderators considered useful include the categorization of user comments in pro- and contra-arguments towards a certain topic, the automated assessment of comments' quality as well as the identification of surprising or exceptional comments and those that present new questions, arguments or viewpoints.
Article
Full-text available
Chat messages of development teams play an increasingly significant role in software development, having replaced emails in some cases. Chat messages contain information about discussed issues, considered alternatives and argumentation leading to the decisions made during software development. These elements, defined as rationale, are invaluable during software evolution for documenting and reusing development knowledge. Rationale is also essential for coping with changes and for effective maintenance of the software system. However, exploiting the rationale hidden in the chat messages is challenging due to the high volume of unstructured messages covering a wide range of topics. This work presents the results of an exploratory study examining the frequency of rationale in chat messages, the completeness of the available rationale and the potential of automatic techniques for rationale extraction. For this purpose, we apply content analysis and machine learning techniques on more than 8,700 chat messages from three software development projects. Our results show that chat messages are a rich source of rationale and that machine learning is a promising technique for detecting rationale and identifying different rationale elements.
Conference Paper
Full-text available
Context: Conducting a Grounded Theory study is rigorous, demanding, and challenging. Misperceptions exist within the software engineering community. Objective: The purpose of this paper is to describe one extended participant observation Grounded Theory study for aiding new empirical researchers wanting to run similar research studies. Method: Following Constructivist Grounded Theory, we conducted a two-year five-month participant-observation of eight software development projects at Pivotal, a software development organization; interviewed 33 software engineers, interaction designers, and product managers; and analyzed one year of retrospection topics. We iterated between analysis and theoretical sampling until achieving theoretical saturation, publishing papers on team code ownership, sustainable software development through overlapping code ownership, and software development waste. Results: This paper describes the missteps, challenges, and unique insights that occurred while conducting a Grounded Theory study. Limitations: While the results are highly relevant to the researcher, the outcomes might not apply to other researchers. Conclusion: Conducting my own Grounded Theory research study, attending Glaser’s Seminar, and reading and rereading Charmaz’s and Glaser’s books helped the researcher overcome misperceptions about Grounded Theory research.
Conference Paper
Reviews for software products contain much information about the users' requirements and preferences, which can be very useful to the requirements engineer. However, taking advantage of this information is not easy due to the large and overwhelming number of reviews that is posted in various channels. Machine learning and opinion mining techniques have therefore been used to process the reviews automatically and to generate summaries of the data to the requirements engineer. However, one of the important challenges for these techniques lies in how to automatically assess the relevance of the reviews for the requirements engineer. So far, most techniques use intuition-based criteria for this task. In this work, we collect and present a list of factors that were found to impact the helpfulness of product reviews for customers. We then discuss to what extent these factors are likely to impact the usefulness of reviews for requirements engineering tasks. The factors can be used to support the automated identification of relevant reviews.
Chapter
One of the challenges in using statistical machine learning for text mining is coming up with the right set of text features. We have developed a system that uses genetic algorithms (GAs) to evaluate candidate feature sets to classify sentences in a document. We have applied this tool to find design rationale (the reasons behind design decisions) in two different datasets to evaluate our approach for finding rationale and to see how features might differ for the same classification target in different types of data. We used Chrome bug reports and transcripts of design sessions. We found that we were able to get results with less overfitting by using a smaller set of features common to the set optimized for each document type.