Carolyn Penstein Rosé

Carnegie Mellon University, Pittsburgh, Pennsylvania, United States

Are you Carolyn Penstein Rosé?

Claim your profile

Publications (191)20.01 Total impact

  • Gahgene Gweon · Soojin Jun · Susan Finger · Carolyn Penstein Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: In project-based learning (PBL) courses, which are common in design and technology education, instructors regard both the process and the final product to be important. However, conducting an accurate assessment for process feedback is not an easy task because instructors of PBL courses often have to make judgments based on a limited view of group work. In this paper, we provide explanations about how in practice instructors actually exhibit cognitive biases and judgments made using incomplete information in the context of an engineering design education classroom. More specifically, we hypothesize that instructors would be susceptible to human errors that are well known in social psychology, the halo effect and the fundamental attribution error, because they have a limited view of group work when they facilitate distributed and remote groups. Through this study, we present two main contributions, namely (1) insights based on classroom data about limitations of current instructor assessment practices, (2) an illustration of using principles from social psychology as a lens for exploring important design questions for designing tools that monitor support oversight of group work. In addition to the study, we illustrate how the findings from our classroom study can be used for effective group assessments.
    International Journal of Technology and Design Education 09/2015; DOI:10.1007/s10798-015-9332-1 · 0.43 Impact Factor
  • Dong Nguyen · A. Seza Doğruöz · Carolyn P. Rosé · Franciska de Jong ·
    [Show abstract] [Hide abstract]
    ABSTRACT: Language is a social phenomenon and inherent to its social nature is that it is constantly changing. Recently, a surge of interest can be observed within the computational linguistics (CL) community in the social dimension of language. In this article we present a survey of the emerging field of "Computational Sociolinguistics" that reflects this increased interest. We aim to provide a comprehensive overview of CL research on sociolinguistic themes, featuring topics such as the relation between language and social identity, language use in social interaction and multilingual communication. Moreover, we demonstrate the potential for synergy between the research communities involved, by showing how the large-scale data-driven methods that are widely used in CL can complement existing sociolinguistic studies, and how sociolinguistics can inform and challenge the methods and assumptions employed in CL studies. We hope to convey the possible benefits of a closer collaboration between the two communities and conclude with a discussion of open challenges.
  • Source
    Marcela Borge · Yann Shiou Ong · Carolyn Penstein Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper we assess the utility of an activity design model and different reflective activities for improving the quality of collaborative processes. Thirty-seven online students, belonging to one of 13 teams, formed the participants of the study. Teams completed five discussion sessions as part of required course activity, using one of two reflective conditions. Each team also received feedback on their performance. We assessed the quality of processes between groups using content analysis techniques. Team process measures at the first time point were used to identify groups' initial strengths and weaknesses. To assess the utility of the model and reflective assessment designs, we used a 2x5 mixed factorial design, with Condition (two levels) as a between subjects factor and Time (5 levels) as a within subjects factor. We found that students were weakest at presenting and discussing claims and both Condition and Time are significant predictors of collaborative process quality.
    The 11th International Conference on Computer Supported Collaborative Learning; 06/2015
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: This interactive event is meant to engage the CSCL community in brainstorming about what affordances in MOOCs would enable application of and research extending theories and best practices from our field. To provide a concrete focus as a foundation for this discussion, we present the innovative design of a recent edX MOOC entitled Data, Analytics, and Learning (DALMOOC). We have integrated several innovative forms of support for discussion based learning, social learning, and self-regulated learning. In particular, we have integrated a layer referred to as ProSolo, which supports social learning and self-directed learning. In further support of self-directed learning, intelligent tutor style exercises have been integrated, which offer immediate feedback and hints to students. We have integrated a social recommendation approach to support effective help seeking in the threaded discussion forums as well as collaborative reflections in the form of synchronous chat exercises facilitated by software agents. The event will include an overview, offering the opportunity for active engagement in the MOOC, structured brainstorming, and interactive, whole group feedback.
    11th International Conference on Computer Supported Collaborative Learning (CSCL 2015), Gothenburg, Swede; 06/2015
  • [Show abstract] [Hide abstract]
    ABSTRACT: An emerging field of educational data mining (EDM) is building on and contributing to a wide variety of disciplines through analysis of data coming from various educational technologies. EDM researchers are addressing questions of cognition, metacognition, motivation, affect, language, social discourse, etc. using data from intelligent tutoring systems, massive open online courses, educational games and simulations, and discussion forums. The data include detailed action and timing logs of student interactions in user interfaces such as graded responses to questions or essays, steps in rich problem solving environments, games or simulations, discussion forum posts, or chat dialogs. They might also include external sensors such as eye tracking, facial expression, body movement, etc. We review how EDM has addressed the research questions that surround the psychology of learning with an emphasis on assessment, transfer of learning and model discovery, the role of affect, motivation and metacognition on learning, and analysis of language data and collaborative learning. For example, we discuss (1) how different statistical assessment methods were used in a data mining competition to improve prediction of student responses to intelligent tutor tasks, (2) how better cognitive models can be discovered from data and used to improve instruction, (3) how data-driven models of student affect can be used to focus discussion in a dialog-based tutoring system, and (4) how machine learning techniques applied to discussion data can be used to produce automated agents that support student learning as they collaborate in a chat room or a discussion board. WIREs Cogn Sci 2015, 6:333-353. doi: 10.1002/wcs.1350 For further resources related to this article, please visit the WIREs website. The authors have declared no conflicts of interest for this article. © 2015 John Wiley & Sons, Ltd.
    Wiley interdisciplinary reviews. Cognitive science 04/2015; 6(4). DOI:10.1002/wcs.1350 · 0.79 Impact Factor
  • D. Yang · M. Wen · I. Howley · R. Kraut · C. Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: Thousands of students enroll in Massive Open Online Courses (MOOCs) to seek opportunities for learning and selfimprovement. However, the learning process often involves struggles with confusion, which may have an adverse effect on the course participation experience, leading to dropout along the way. In this paper, we quantify that effect. We describe a classification model using discussion forum behavior and clickstream data to automatically identify posts that express confusion. We then apply survival analysis to quantify the impact of confusion on student dropout. The results demonstrate that the more confusion students express or are exposed to, the lower the probability of their retention. Receiving support and resolution of confusion helps mitigate this effect. We explore the differential effects of confusion expressed in different contexts and related to different aspects of courses. We conclude with implications for design of interventions towards improving the retention of students in MOOCs.
  • O. Ferschke · G. Tomar · C.P. Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper we explore how to import intelligent support for group learning that has been demonstrated as effective in classroom instruction into a Massive Open Online Course (MOOC) context. The Bazaar agent architecture paired with an innovative Lobby tool to enable coordination for synchronous reflection exercises provides a technical foundation for our work. We describe lessons learned, directions for future work, and offer pointers to resources for other researchers interested in computer supported collaborative learning in MOOCs.
  • Source
    Diyi Yang · Miaomiao Wen · Abhimanu Kumar · Eric P. Xing · Carolyn Penstein Rose ·
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper, we describe a novel methodology, grounded in techniques from the field of machine learning, for modeling emerging social structure as it develops in threaded discussion forums, with an eye towards application in the threaded discussions of massive open online courses (MOOCs). This modeling approach integrates two simpler, well established prior techniques, namely one related to social network structure and another related to thematic structure of text. As an illustrative application of the integrated technique's use and utility, we use it as a lens for exploring student dropout behavior in three different MOOCs. In particular, we use the model to identify twenty emerging subcommunities within the threaded discussions of each of the three MOOCs. We then use a survival model to measure the impact of participation in identified subcommunities on attrition along the way for students who have participated in the course discussion forums of the three courses. In each of three MOOCs we find evidence that participation in two to four subcommunities out of the twenty is associated with significantly higher or lower dropout rates than average. A qualitative post-hoc analysis illustrates how the learned models can be used as a lens for understanding the values and focus of discussions within the subcommunities, and in the illustrative example to think about the association between those and detected higher or lower dropout rates than average in the three courses. Our qualitative analysis demonstrates that the patterns that emerge make sense: It associates evidence of stronger expressed motivation to actively participate in the course as well as evidence of stronger cognitive engagement with the material in subcommunities associated with lower attrition, and the opposite in subcommunities associated with higher attrition. We conclude with a discussion of ways the modeling approach might be applied, along with caveats from limitations, and directions for future work.
    International Review of Research in Open and Distance Learning 11/2014; 15(5):214-234. · 0.69 Impact Factor
  • Source
    Miaomiao Wen · Diyi Yang · Carolyn Penstein Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: While data from Massive Open Online Courses (MOOCs) offers the potential to gain new insights into the ways in which online communities can contribute to student learning, much of the richness of the data trace is still yet to be mined. In particular, very little work has attempted fine-grained content analyses of the student interactions in MOOCs. Survey research indicates the importance of student goals and intentions in keeping them involved in a MOOC over time. Automated fine-grained content analyses offer the potential to detect and monitor evidence of student engagement and how it relates to other aspects of their behavior. Ultimately these indicators reflect their commitment to remaining in the course. As a methodological contribution, in this paper we investigate using computational linguistic models to measure learner motivation and cognitive engagement from the text of forum posts. We validate our techniques using survival models that evaluate the predictive validity of these variables in connection with attrition over time. We conduct this evaluation in three MOOCs focusing on very different types of learning materials. Prior work demonstrates that participation in the discussion forums at all is a strong indicator of student commitment. Our methodology allows us to differentiate better among these students, and to identify danger signs that a struggling student is in need of support within a population whose interaction with the course offers the opportunity for effective support to be administered. Theoretical and practical implications will be discussed.
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Learning analytics (LA) as a field remains in its infancy. Many of the techniques now prominent from practitioners have been drawn from various fields, including HCI, statistics, computer science, and learning sciences. In order for LA to grow and advance as a discipline, two significant challenges must be met: 1) development of analytics methods and techniques that are native to the LA discipline, and 2) practitioners in LA to develop algorithms and models that reflect the social and computational dimensions of analytics. This workshop introduces researchers in learning analytics to machine learning (ML) and the opportunities that ML can provide in building next generation analysis models.
    Proceedings of the Fourth International Conference on Learning Analytics And Knowledge; 03/2014
  • Source
    Carolyn Penstein Rosé · Ryan Carlson · Diyi Yang · Miaomiao Wen · Lauren Resnick · Pam Goldman · Jennifer Sherer ·
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper, we explore student dropout behavior in a Massively Open Online Course (MOOC). We use a survival model to measure the impact of three social factors that make predictions about attrition along the way for students who have participated in the course discussion forum.
    Proceedings of the first ACM conference on Learning @ scale conference; 03/2014
  • Iris Howley · Takayuki Kanda · Kotaro Hayashi · Carolyn Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: The unique social presence of robots can be leveraged in learning situations to reduce student evaluation anxiety, while still providing instructional guidance on multiple levels of communication. Furthermore, social role of the instructor can also impact the prevalence of evaluation apprehension. In this study, we examine how human and robot social role affects help-seeking behaviors and learning outcomes in a one-on-one tutoring setting. Our results show that help-seeking is a moderator of the significant relationship between condition and learning, with the "human teacher" condition resulting in significantly less learning (and marginally less help-seeking) than the "human assistant" and both robot conditions.
    Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction; 03/2014
  • David Adamson · Gregory Dyke · Hyeju Jang · Carolyn Penstein Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: This paper investigates the use of conversational agents to scaffold on-line collaborative learning discussions through an approach called Academically Productive Talk (APT). In contrast to past work on dynamic support for collaborative learning, where agents were used to elevate conceptual depth by leading students through directed lines of reasoning (Kumar & Rosé, IEEE Transactions on Learning Technologies, 4(1), 2011), this APT-based approach uses generic prompts that encourage students to articulate and elaborate their own lines of reasoning, and to challenge and extend the reasoning of their teammates. This paper integrates findings from a series of studies across content domains (biology, chemistry, engineering design), grade levels (high school, undergraduate), and facilitation strategies. APT based strategies are contrasted with simply offering positive feedback when the students themselves employ APT facilitation moves in their interactions with one another, an intervention we term Positive Feedback for APT engagement. The pattern of results demonstrates that APT based support for collaborative learning can significantly increase learning, but that the effect of specific APT facilitation strategies is context specific. It appears the effectiveness of each strategy depends upon factors such as the difficulty of the material (in terms of being new conceptual material versus review) and the skill level of the learner (urban public high school vs. selective private university). In contrast, Feedback for APT engagement does not positively impact learning. In addition to an analysis based on learning gains, an automated conversation analysis technique is presented that effectively predicts which strategies are successfully operating in specific contexts. Implications for design of more agile forms of dynamic support for collaborative learning are discussed.
    01/2014; 24(1):92-124. DOI:10.1007/s40593-013-0012-6
  • Rohit Kumar · Carolyn P. Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: Conversational agent technology is an emerging paradigm for creating a social environment in online groups that is conducive to effective teamwork. Prior work has demonstrated advantages in terms of learning gains and satisfaction scores when groups learning together online have been supported by conversational agents that employ Balesian social strategies. This prior work raises two important questions that are addressed in this article. The first question is one of generality. Specifically, are the positive effects of the designed support specific to learning contexts? Or are they in evidence in other collaborative task domains as well? We present a study conducted within a collaborative decision-making task where we see that the positive effects of the Balesian social strategies extend to this new context. The second question is whether it is possible to increase the effectiveness of the Balesian social strategies by increasing the context sensitivity with which the social strategies are triggered. To this end, we present technical work that increases the sensitivity of the triggering. Next, we present a user study that demonstrates an improvement in performance of the support agent with the new, more sensitive triggering policy over the baseline approach from prior work. The technical contribution of this article is that we extend prior work where such support agents were modeled using a composition of conversational behaviors integrated within an event-driven framework. Within the present approach, conversation is orchestrated through context-sensitive triggering of the composed behaviors. The core effort involved in applying this approach involves building a set of triggering policies that achieve this orchestration in a time-sensitive and coherent manner. In line with recent developments in data-driven approaches for building dialog systems, we present a novel technique for learning behavior-specific triggering policies, deploying it as part of our efforts to improve a socially capable conversational tutor agent that supports collaborative learning.
    Transactions on Interactive Intelligent Systems 01/2014; 3(4). DOI:10.1145/2499672
  • Source
    M Wen · D Yang · C P Rosé ·

    Educational Data Mining; 01/2014
  • Source
    Diyi Yang · Tanmay Sinha · David Adamson · Carolyn Penstein Rose ·
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper, we explore student dropout behavior in Massive Open Online Courses(MOOC). We use as a case study a recent Coursera class from which we develop a survival model that allows us to measure the influence of factors extracted from that data on student dropout rate. Specifically we explore factors related to student behavior and social positioning within discussion forums using standard social network analytic techniques. The analysis reveals several significant predictors of dropout.
    NIPS Workshop on Data Driven Education; 12/2013
  • Source
    Elijah Mayfield · M Barton Laws · Ira B Wilson · Carolyn Penstein Rose ·
    [Show abstract] [Hide abstract]
    ABSTRACT: Coding of clinical communication for fine-grained features such as speech acts has produced a substantial literature. However, annotation by humans is laborious and expensive, limiting application of these methods. We aimed to show that through machine learning, computers could code certain categories of speech acts with sufficient reliability to make useful distinctions among clinical encounters. The data were transcripts of 415 routine outpatient visits of HIV patients which had previously been coded for speech acts using the Generalized Medical Interaction Analysis System (GMIAS); 50 had also been coded for larger scale features using the Comprehensive Analysis of the Structure of Encounters System (CASES). We aggregated selected speech acts into information-giving and requesting, then trained the machine to automatically annotate using logistic regression classification. We evaluated reliability by per-speech act accuracy. We used multiple regression to predict patient reports of communication quality from post-visit surveys using the patient and provider information-giving to information-requesting ratio (briefly, information-giving ratio) and patient gender. Automated coding produces moderate reliability with human coding (accuracy 71.2%, κ=0.57), with high correlation between machine and human prediction of the information-giving ratio (r=0.96). The regression significantly predicted four of five patient-reported measures of communication quality (r=0.263-0.344). The information-giving ratio is a useful and intuitive measure for predicting patient perception of provider-patient communication quality. These predictions can be made with automated annotation, which is a practical option for studying large collections of clinical encounters with objectivity, consistency, and low cost, providing greater opportunity for training and reflection for care providers.
    Journal of the American Medical Informatics Association 09/2013; 21(E1). DOI:10.1136/amiajnl-2013-001898 · 3.50 Impact Factor
  • Elijah Mayfield · David Adamson · Carolyn Penstein Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: Automated annotation of social behavior in conversation is necessary for large-scale analysis of real-world conversational data. Important behavioral categories, though, are often sparse and often appear only in specific subsections of a conversation. This makes supervised machine learning difficult, through a combination of noisy features and unbalanced class distributions. We propose within-instance content selection, using cue features to selectively suppress sections of text and biasing the remaining representation towards minority classes. We show the effectiveness of this technique in automated annotation of empowerment language in online support group chatrooms. Our technique is significantly more accurate than multiple baselines, especially when prioritizing high precision.
    Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); 08/2013
  • Source
    Miaomiao Wen · Zeyu Zheng · Hyeju Jang · Guang Xiang · Carolyn Penstein Rosé ·
    [Show abstract] [Hide abstract]
    ABSTRACT: We present a system for extracting the dates of illness events (year and month of the event occurrence) from posting histories in the context of an online medical support community. A temporal tagger retrieves and normalizes dates mentioned informally in social media to actual month and year referents. Building on this, an event date extraction system learns to integrate the likelihood of candidate dates extracted from time-rich sentences with temporal constraints extracted from event-related sentences. Our integrated model achieves 89.7% of the maximum performance given the performance of the temporal expression retrieval step.
    Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers); 08/2013
  • Gregory Dyke · David Adamson · Iris Howley · C.P. Rose ·
    [Show abstract] [Hide abstract]
    ABSTRACT: This paper investigates the use of conversational agents to scaffold online collaborative learning discussions through an approach called academically productive talk (APT). In contrast to past work on dynamic support for collaborative learning, which has involved using agents to elevate the conceptual depth of collaborative discussion by leading students in groups through directed lines of reasoning, this APT-based approach lets students follow their own lines of reasoning and promotes productive practices such as explanation of reasoning and refinement of ideas. Two forms of support are contrasted, namely, Revoicing support and Feedback support. The study provides evidence that Revoicing support resulted in significantly more intensive reasoning exchange between students in the chat and significantly more learning during the chat than when that form of support was absent. Another form of support, namely, Feedback support increased expression of reasoning while marginally decreasing the intensity of the interaction between students and did not affect learning.
    IEEE Transactions on Learning Technologies 07/2013; 6(3):240-247. DOI:10.1109/TLT.2013.25 · 1.28 Impact Factor

Publication Stats

2k Citations
20.01 Total Impact Points


  • 2-2015
    • Carnegie Mellon University
      • • Human-Computer Interaction Institute
      • • Language Technologies Institute
      • • Computer Science Department
      Pittsburgh, Pennsylvania, United States
  • 1998-2006
    • University of Pittsburgh
      • Learning Research and Development Center
      Pittsburgh, Pennsylvania, United States