Maciej Pankiewicz

University of Pennsylvania | UP · Penn Center for Learning Analytics

Addressing the challenge of generating personalized feedback for programming assignments is demanding due to several factors, like the complexity of code syntax or different ways to correctly solve a task. In this experimental study, we automated the process of feedback generation by employing OpenAI's GPT-3.5 model to generate personalized hints f...

University-level computer science (CS) courses can be particularly challenging for students with limited programming backgrounds. To support novice learners, instructors often employ automated assessment systems for programming assignments. These systems provide students with feedback on demand as they work through problems online. Outcomes remain...

In this paper, we discuss the application of JeepyTA-a virtual teaching assistant (TA) powered by a large language model (LLM)-within a graduate-level course on games and learning in Spring 2024. Specifically, we describe how JeepyTA was integrated in two "play journal" assignments where twenty-two students documented their observations of playing...

This paper examines the advantages and limitations of conducting automated coding of virtual tutoring session transcripts using the GPT-4 Turbo model via the OpenAI API. We compare three coding methods: (1) zero-shot, which relies solely on construct definitions; (2) few-shot, which includes annotated examples; and (3) coding with context, which pr...

As student learning transitions to being increasingly 24/7, online courses struggle to provide support for learners on the same schedule. Human TAs are bound by time constraints and are often available only during limited working hours. This can result in a wait time for students seeking answers to their coursework questions. This paper introduces...

Education is increasingly taking place in learning environments mediated by technology. This transition has made it easier to collect student-generated data including comments in discussion forums and chats. Although this data is extremely valuable to researchers, it often contains sensitive information like names, locations, social media links, an...

We examined the efficacy of AI-assisted learning in an introductory programming course at the university level by using a GPT-4 model to generate personalized hints for compiler errors within a platform for automated assessment of programming assignments. The control group had no access to GPT hints. In the experimental condition GPT hints were pro...

Computer science (CS) education at the university level is often challenging, particularly for students with no prior programming experience. To help scaffold students’ CS learning, instructors often utilize systems for automated assessment of programming assignments, where students can individually learn online using automatically generated feedba...

Programming courses can be challenging for first year university students, especially for those without prior coding experience. Students initially struggle with code syntax, but as more advanced topics are introduced across a semester, the difficulty in learning to program shifts to learning computational thinking (e.g., debugging strategies). Thi...

Self-regulated learning (SRL) is important for computer science education. Yet, students often do not have SRL skills to benefit their learning. In this study, we examined 187 (n=187) students' SRL behaviors while they built programs with an automated feedback tool. Anchored in Winne and Hadwin's (1998) COPES model of SRL, our results showed that n...

There exist several online applications for automated testing of the computer programs that students write in computer science education. Use of such systems enables self-paced learning with automated feedback delivered by the application. However, due to the complexity of programming languages, even the easiest tasks made available through such sy...

There exist several online applications for automated testing of the computer programs that students write in computer science education. Use of such systems enables self-paced learning with automated feedback delivered by the application. However, due to the complexity of programming languages, even the easiest tasks made available through such sy...

Addressing the challenge of generating personalized feedback for programming assignments is demanding due to several factors, like the complexity of code syntax or different ways to correctly solve a task. In this experimental study, we automated the process of feedback generation by employing OpenAI's GPT-3.5 model to generate personalized hints f...

This study presents a comparison of the following methods for estimating task difficulty in terms of the so-called "cold start" problem - during the initial phase of the introduction of the adaptive educational system to the public. The data originates from the item-based online programming course made available on the RunCode online learning platf...

There are several methods aimed at assessing assignment difficulty and learner ability that originate to a great extent from the area of item response theory (IRT). Computational demands have been defined as the main hurdle in the operational usage of these models within adaptive online learning environments. Therefore, alternative methods of diffi...

The aim of this study is to present and evaluate the Elo rating algorithm as a tool for assessing the task difficulty in terms of the so-called "cold-start" problem-during the initial phase of the introduction of the adaptive system to the public. This analysis has been performed on the real data originating from the online programming course avail...

The aim of this research is to examine the accuracy of the estimations performed with the Elo rating system in an online learning environment where multiple attempts are allowed and feedback is provided after every submission. The acquired estimations are compared to the reference difficulty values calculated by the means of the IRT graded response...

One of the features of online learning environments is an option of an automated evaluation of programming tasks. This feature, combined with immediate feedback after every submission may support the self-progress of students in the context of programming classes. Such a system has been introduced as an additional tool within the university's Intro...

In this research, we analyze data from five editions of the Introduction to programming course held between academic years 2014/2015 and 2018/2019 at the Faculty of Applied Informatics and Mathematics, Warsaw University of Life Sciences. There have been 969 students that took the final exam during analyzed period. There are two sources of data: 1)...

The Elo rating algorithm, developed for the purpose of measuring player strength in chess tournaments, has also found application in the context of educational research and has been used for the purpose of measuring both learner ability and task difficulty. The quality of the estimations performed by the Elo rating algorithm has already been subjec...

Images of natural scenes, like those relevant for agriculture, are characterised with a variety of forms of objects of interest and similarities between objects that one might want to discriminate. This introduces uncertainty to the analysis of such images. Requirements for an image annotation tool to be used in pattern recognition design for agric...

It has been observed, that interest of researchers in the subject of learning process gamification increases. The number of publications presenting results of gamification experiments in educational environments is growing. These results contribute to the research area of influence of gamification on the level of the engagement and motivation of pa...

The algorithms of color transformation that preserves the impression of texture are used in virtual makeover systems, where maintaining the impression of unaltered texture is important in the process of transforming the color. The content of this paper covers the process of implementing the algorithm of digital picture color transformation with its...

This paper will discuss the ongoing research in area of conceptualization, design of collaborative working environments and its adoption for food supply networks. There will be a group needs approach for designing such environments presented. Tracking and tracing, agreements on quality policy, improvements in logistics, dissemination of innovations...

One of the critical success factors for the future development of the agri-food sector is the increased integration of enterprises and other stakeholders in horizontal and vertical ‘communities’. Tracking and tracing, agreements on quality policy, improvements in logistics, dissemination of innovations, cooperation in quality planning and market or...