Dominik Gorgosch’s research while affiliated with Chemnitz University of Technology and other places

What is this page?


This page lists works of an author who doesn't have a ResearchGate profile or hasn't added the works to their profile yet. It is automatically generated from public (personal) data to further our legitimate goal of comprehensive and accurate scientific recordkeeping. If you are this author and want this page removed, please let us know.

Publications (1)


Different prompt purposes of programming beginners with a conversational chatbot
Study procedure. Participants were assigned to either the experimental or control condition and completed 7 tasks over a period of 7 weeks
Structure of results analysis. Data was analyzed on three different levels, beginning with individual prompt purposes to intentions of conversations
Distribution of task performance between participants in the experiment group (“Chatbot”, light red) and the control group (“Control”, gray)
Distribution of prompt purposes by task

+1

“Ok Pal, we have to code that now”: interaction patterns of programming beginners with a conversational chatbot
  • Article
  • Publisher preview available

November 2024

·

20 Reads

·

1 Citation

Empirical Software Engineering

·

Dominik Gorgosch

·

·

Context Chatbots based on large language models are becoming an important tool in modern software development, yet little is known about how programming beginners interact with this new technology to write code and acquire new knowledge. Thus, we are missing key ingredients to develop guidelines on how to adopt chatbots for becoming productive at programming. Objective With our research, we aim at identifying these ingredients. Specifically, we want to understand how programming beginners use conversational chatbots when writing source code. Method To this end, we study programming beginners’ interaction with a chatbot in a CS2 course while they were solving programming assignments. Additionally, we evaluate the correctness of submitted solutions and compare them to solutions of beginners who did not use a conversational chatbot. Results We analyzed 756 prompts of 129 conversations, most of them focusing on code generation. Interestingly, conversations that contain prompts asking for debugging or testing of code are linked with higher success rates, indicating that deeper engagement with code leads to higher quality code. Moreover, prompts without sufficient context often lead to unsatisfying results. Conclusions While not surprising, this underpins the importance that programming beginners need to know how to use chatbots, instead of merely using it as code generators without investing time in code quality. Moreover, companies should employ prompt guidelines, in which code quality prompts might be enforced after a code generation prompt has been stated.

View access options

Citations (1)


... Thus the initial research in the area suggests that novices struggle to directly use LLMs in an effective manner. This chimes with other research considering the pedagogical implications: Xue et al. [82] and Kazemitabaar et al. [32] found that direct use of LLMs did not produce any significant effect on learning (although the latter suggest that students with higher prior knowledge may have received greater benefits from using the generator than students with less prior knowledge), while Mailach et al. [49] concluded that "we cannot just give vanilla [LLM] chatbots to students as tools to learn programming, but we additionally need to give proper guidance on how to use them-otherwise, students tend to use it mainly for code generation without further reflection on or evaluation of generated code. " 2.3.2 ...

Reference:

Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers
“Ok Pal, we have to code that now”: interaction patterns of programming beginners with a conversational chatbot

Empirical Software Engineering