No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Analysis of Novices' Web-Based Help-Seeking Behavior While Programming
... However, traditionally, learners proactively searched for and found relevant code examples to address gaps in their knowledge, meaningfully engaging themselves in a learning process [10,15,70,101]. But with LLMs, learners can bypass the effortful yet beneficial process of adapting generic code examples to specific contexts [55,112]. ...
Novice programmers are increasingly relying on Large Language Models (LLMs) to generate code for learning programming concepts. However, this interaction can lead to superficial engagement, giving learners an illusion of learning and hindering skill development. To address this issue, we conducted a systematic design exploration to develop seven cognitive engagement techniques aimed at promoting deeper engagement with AI-generated code. In this paper, we describe our design process, the initial seven techniques and results from a between-subjects study (N=82). We then iteratively refined the top techniques and further evaluated them through a within-subjects study (N=42). We evaluate the friction each technique introduces, their effectiveness in helping learners apply concepts to isomorphic tasks without AI assistance, and their success in aligning learners' perceived and actual coding abilities. Ultimately, our results highlight the most effective technique: guiding learners through the step-by-step problem-solving process, where they engage in an interactive dialog with the AI, prompting what needs to be done at each stage before the corresponding code is revealed.
... However, for programming novices, both approaches have potential pitfalls. Web searches can lead to the faulty integration of poorly-understood code, resulting in compounding errors [25]. ...
Growth in the use of large language models (LLMs) in programming education is altering how students write SQL queries. Traditionally, students relied heavily on web search for coding assistance, but this has shifted with the adoption of LLMs like ChatGPT. However, the comparative process and outcomes of using web search versus LLMs for coding help remain underexplored. To address this, we conducted a randomized interview study in a database classroom to compare web search and LLMs, including a publicly available LLM (ChatGPT) and an instructor-tuned LLM, for writing SQL queries. Our findings indicate that using an instructor-tuned LLM required significantly more interactions than both ChatGPT and web search, but resulted in a similar number of edits to the final SQL query. No significant differences were found in the quality of the final SQL queries between conditions, although the LLM conditions directionally showed higher query quality. Furthermore, students using instructor-tuned LLM reported a lower mental demand. These results have implications for learning and productivity in programming education.
... Software developers often resort to online resources for a variety of software engineering tasks, e.g., API learning, bug fixing, comprehension of code or concepts, etc. [53,57,63]. A vast majority of these help-seeking activities include frequent engagement with community Q&A platforms such as Stack Overflow 1 (SO) [52,53,62,63] to seek help, solutions, or suggestions from other developers. ...
Q&A platforms have been an integral part of the web-help-seeking behavior of programmers over the past decade. However, with the recent introduction of ChatGPT, the paradigm of web-help-seeking behavior is experiencing a shift. Despite the popularity of ChatGPT, no comprehensive study has been conducted to evaluate the characteristics or usability of ChatGPT's answers to software engineering questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT's answers to 517 Stack Overflow (SO) questions and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT's answers. Furthermore, we conducted a large-scale linguistic analysis, and a user study to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52\% of ChatGPT answers are incorrect and 77\% are verbose. Nonetheless, ChatGPT answers are still preferred 39.34\% of the time due to their comprehensiveness and well-articulated language style. Our result implies the necessity of close examination and rectification of errors in ChatGPT, at the same time creating awareness among its users of the risks associated with seemingly correct ChatGPT answers.
Solving complex Computer Science problems require students to utilize both higher level skills like problem translation, decomposition, and implementation, and lower level skills such as implementing solution patterns and debugging. However, novices often struggle with with these lower level skills, preventing them from moving forward. As a consequence, they risk developing a shallow understanding of code concepts and fail to practice crucial higher-level skills. We assert that if students were given the ability to practice these lower level skills outside of traditional programming exercises, they would be more apt to recognize and resolve them. For this we present our analysis of TYPOS, a platform for low-level skill practice using interactive worked examples as typing exercises. In this work we analyzed self-selected students based on their platform usage and the differences in course performance and behavior between groups. We found that students in a college level CS course that regularly completed typing exercises earned higher course grades and higher learning gains than those who worked irregularly or not at all. Students who actively used the system and completed typing exercises made fewer compilation errors on graded projects. Students did not, however, use the tool while working on their projects but engaged in deliberate practice sessions prior to lecture. We argue this model provides students with the benefits (although with marginal significance) that accrue from worked examples as well as deliberate practice on a key area of struggle for novice students. CCS CONCEPTS • Theory of computation → Active learning; • Applied computing → Interactive learning environments.
Tinkering has been shown to have a positive influence on students in open-ended making activities. Open-ended programming assignments in block-based programming resemble making activities in that both of them encourage students to tinker with tools to create their own solutions to achieve a goal. However, previous studies of tinkering in programming discussed tinkering as a broad, ambiguous term, and investigated only self-reported data. To our knowledge, no research has studied student tinkering behaviors while solving problems in block-based programming environments. In this position paper, we propose a definition for tinkering in block-based programming environments as a kind of behavior that students exhibit when testing, exploring, and struggling during problem-solving. We introduce three general categories of tinkering behaviors (test-based, prototype-based, and construction-based tinkering) derived from student data, and use case studies to demonstrate how students exhibited these behaviors in problem-solving. We created the definitions using a mixed-methods research design combining a literature review with data-driven insights from submissions of two open-ended programming assignments in iSnap, a block-based programming environment. We discuss the implication of each type of tinkering behavior for learning. Our study and results are the first in this domain to define tinkering based on student behaviors in a block-based programming environment.
Technical question and answer Q&A platforms, such as Stack Overflow, provide a platform for users to ask and answer questions about a wide variety of programming topics. These platforms accumulate a large amount of knowledge, including hundreds of thousands lines of source code. Developers can benefit from the source code that is attached to the questions and answers on Q&A platforms by copying or learning from (parts of) it. By understanding how developers utilize source code from Q&A platforms, we can provide insights for researchers which can be used to improve next-generation Q&A platforms to help developers reuse source code fast and easily. In this paper, we first conduct an exploratory study on 289 files from 182 open-source projects, which contain source code that has an explicit reference to a Stack Overflow post. Our goal is to understand how developers utilize code from Q&A platforms and to reveal barriers that may make code reuse more difficult. In 31.5% of the studied files, developers needed to modify source code from Stack Overflow to make it work in their own projects. The degree of required modification varied from simply renaming variables to rewriting the whole algorithm. Developers sometimes chose to implement an algorithm from scratch based on the descriptions from Stack Overflow answers, even if there was an implementation readily available in the post. In 35.5% of the studied files, developers used Stack Overflow posts as an information source for later reference. To further understand the barriers of reusing code and to obtain suggestions for improving the code reuse process on Q&A platforms, we conducted a survey with 453 open-source developers who are also on Stack Overflow. We found that the top 3 barriers that make it difficult for developers to reuse code from Stack Overflow are: (1) too much code modification required to t in their projects, (2) incomprehensive code, and (3) low code quality. We summarized and analyzed all survey responses and we identified that developers suggest improvements for future Q&A platforms along the following dimensions: code quality, information enhancement & management, data organization, license, and the human factor. For instance, developers suggest to improve the code quality by adding an integrated validator that can test source code online, and an outdated code detection mechanism. Our findings can be used as a roadmap for researchers and developers to improve code reuse.
When novice students encounter diculty when learning to program, some can seek help from instructors or teaching assistants. is one-on-one tutoring is highly eective at fostering learning, but busy instructors and large class sizes can make expert help a scarce resource. Increasingly, programming environments attempt to imitate this human support by providing students with hints and feedback. In order to design eective, computer-based help, it is important to understand how and why students seek and avoid help when programming, and how this process diers when the help is provided by a human or a computer. We explore these questions through a qualitative analysis of 15 students’ interviews, in which they reect on solving two programming problems with human and computer help. We discuss implications for help design and present hypotheses on students’ help-seeking behavior.
In this paper, we investigate the relationship between searching and learning, by conceptualizing information seeking as a learning process, and learning as an outcome of the information seeking process. We present the participants with four search tasks, each of them designed to represent different cognitive levels of learning. Through quantitative analysis of the participants» Web search logs, we examine how individual search behavior is influenced by different task complexity levels as we present the tasks in a hierarchical order. We also explore how the perceived learning outcomes and processes, and the different learning actions, are related to the levels of cognitive complexity. By analyzing the search logs, self-reports, interview data, and the reports, both quantitatively and qualitatively, we infer that searching and learning are not isolated but co-existing processes. Distinct search patterns and learning outcomes were observed in tasks of different cognitive complexities, and overlapping learning actions were observed for the different tasks.
Developers commonly make use of a web search engine such as Google to locate online resources to improve their productivity. A better understanding of what developers search for could help us understand their behaviors and the problems that they meet during the software development process. Unfortunately, we have a limited understanding of what developers frequently search for and of the search tasks that they often find challenging. To address this gap, we collected search queries from 60 developers, surveyed 235 software engineers from more than 21 countries across five continents. In particular, we asked our survey participants to rate the frequency and difficulty of 34 search tasks which are grouped along the following seven dimensions: general search, debugging and bug fixing, programming, third party code reuse, tools, database, and testing. We find that searching for explanations for unknown terminologies, explanations for exceptions/error messages (e.g., HTTP 404), reusable code snippets, solutions to common programming bugs, and suitable third-party libraries/services are the most frequent search tasks that developers perform, while searching for solutions to performance bugs, solutions to multi-threading bugs, public datasets to test newly developed algorithms or systems, reusable code snippets, best industrial practices, database optimization solutions, solutions to security bugs, and solutions to software configuration bugs are the most difficult search tasks that developers consider. Our study sheds light as to why practitioners often perform some of these tasks and why they find some of them to be challenging. We also discuss the implications of our findings to future research in several research areas, e.g., code search engines, domain-specific search engines, and automated generation and refinement of search queries.
Many software development organizations strive to enhance the productivity of their developers. All too often, efforts aimed at improving developer productivity are undertaken without knowledge about how developers spend their time at work and how it influences their own perception of productivity. To fill in this gap, we deployed a monitoring application at 20 computers of professional software developers from four companies for an average of 11 full workdays in situ. Corroborating earlier findings, we found that developers spend their time on a wide variety of activities and switch regularly between them, resulting in highly fragmented work. Our findings extend beyond existing research in that we correlate developers’ work habits with perceived productivity and also show productivity is a personal matter. Although productivity is personal, developers can be roughly grouped into morning, low-at-lunch and afternoon people. A stepwise linear regression per participant revealed that more user input is most often associated with a positive, and emails, planned meetings and work unrelated websites with a negative perception of productivity. We discuss opportunities of our findings, the potential to predict high and low productivity and suggest design approaches to create better tool support for planning developers’ workdays and improving their personal productivity.
Software development often requires knowledge beyond what developers already possess. In such cases, developers have to seek help from different sources of information. As a metacognitive skill, help seeking influences software developers' efficiency and success in many situations. However, there has been little research to provide a systematic investigation of the general process of help seeking activities in software engineering and human and system factors affecting help seeking. This paper reports our empirical study aiming to fill this gap. Our study includes two human experiments, involving 24 developers and two typical software development tasks. Our study gathers empirical data that allows us to provide an in-depth analysis of help-seeking task structures, task strategies, information sources, process model, and developers' information needs and behaviors in seeking and using help information and in managing information during help seeking. Our study provides a detailed understanding of help seeking activities in software engineering, the challenges that software developers face, and the limitations of existing tool support. This can lead to the design and development of more efficient and usable help seeking support that helps developers become better help seekers.
Thematic analysis is a poorly demarcated, rarely acknowledged, yet widely used qualitative analytic method within psychology. In this paper, we argue that it offers an accessible and theoretically flexible approach to analysing qualitative data. We outline what thematic analysis is, locating it in relation to other qualitative analytic methods that search for themes or patterns, and in relation to different epistemological and ontological positions. We then provide clear guidelines to those wanting to start thematic analysis, or conduct it in a more deliberate and rigorous way, and consider potential pitfalls in conducting thematic analysis. Finally, we outline the disadvantages and advantages of thematic analysis. We conclude by advocating thematic analysis as a useful and flexible method for qualitative research in and beyond psychology.
Two studies examined associations between college students' help seeking and perceptions of their classes' achievement goal structure. Study 1 established that students' help seeking (N = 883 in 6 chemistry classes) could be parsimoniously described by distinct approach (intentions to seek autonomous help from teachers) and avoidance patterns (threat, avoidance intentions, seeking expedient help). In Study 2, after controlling for students' personal achievement goal orientations (N = 852 in 13 psychology classes), within-class differences in perceived class emphasis on mastery positively predicted help-seeking approach and negatively predicted help-seeking avoidance patterns, whereas perceived class emphasis on performance-avoid goals positively predicted help-seeking avoidance. Students in classes with greater perceived emphasis on performance-avoid goals had higher levels of help-seeking avoidance patterns. Results complement previous research on help seeking and achievement goals with younger learners and provide support for the role of classroom achievement goal structure in student motivation and performance. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
The practice of guiding a search engine based on query logs observed from the engine's user population provides large volumes of data but potentially also sacrifices the privacy of the user. In this paper, we ask the following question: Is it possible, given rich instrumented data from a panel and usability study data, to observe complete information without routinely analyzing query logs? What unique benefits to the user could hypothetically be derived from analyzing query logs? We demonstrate that three different modes of collecting data, the field study, the instrumented user panel, and the raw query log, provide complementary sources of data. The query log is the least rich source of data for individual events, but has irreplaceable information for understanding the scope of resources that a search engine needs to provide for the user.
This paper investigates the role of online resources in prob- lem solving. We look specifically at how programmers—an exemplar form of knowledge workers—opportunistically in- terleave Web foraging, learning, and writing code. We de- scribe two studies of how programmers use online resources. The first, conducted in the lab, observed participants' Web use while building an online chat room. We found that pro- grammers leverage online resources with a range of inten- tions: They engage in just-in-time learning of new skills and approaches, clarify and extend their existing knowledge, and remind themselves of details deemed not worth remember- ing. The results also suggest that queries for different pur- poses have different styles and durations. Do programmers' queries "in the wild" have the same range of intentions, or is this result an artifact of the particular lab setting? We an- alyzed a month of queries to an online programming portal, examining the lexical structure, refinements made, and re- sult pages visited. Here we also saw traits that suggest the Web is being used for learning and reminding. These results contribute to a theory of online resource usage in program- ming, and suggest opportunities for tools to facilitate online knowledge work. Author Keywords
Why do some students avoid seeking help in the classroom when they need it? When students do not seek the help they need, they put themselves at a disadvantage for learning. We discuss how students' personal motivational characteristics relate to their avoidance of help seeking. In particular, we discuss our work regarding perceived academic and social competence and achievement- and social-goal orientations. We also discuss how various dimensions of the classroom context relate to help avoidance. Specifically, we discuss rules and norms of classrooms as well as our work examining the achievement goal structure and social climate of classrooms. We conclude by discussing how new developments in achievement goal theory distinguishing between approach- and avoidance-goal orientations might add to psychologists' understanding of help avoidance. We also consider how students' strivings for autonomy and different aspects of the social climate might be incorporated into theories of students' help avoidance behavior. Peer Reviewed http://deepblue.lib.umich.edu/bitstream/2027.42/44450/1/10648_2004_Article_292341.pdf
While programming, novices often lack the ability to effectively seek help, such as when to ask for a hint or feedback. Students may avoid help when they need it, or abuse help to avoid putting in effort, and both behaviors can impede learning. In this paper we present two main contributions. First, we investigated log data from students working in a programming environment that offers automated hints, and we propose a taxonomy of unproductive help-seeking behaviors in programming. Second, we used these findings to design a novel user interface for hints that subtly encourages students to seek help with the right frequency, estimated with a data-driven algorithm. We conducted a pilot study to evaluate our data-driven (DD) hint display, compared to a traditional interface, where students request hints on-demand as desired. We found students with the DD display were less than half as likely to engage in unproductive help-seeking, and we found suggestive evidence that this may improve their learning. The primary contributions of this work are a novel taxonomy of unproductive help-seeking behaviors in programming and design insights that suggest how help interfaces can deter this behavior.
Monthly, 50 million users visit Stack Overflow, a popular Q&A forum used by software developers, to share and gather knowledge and help with coding problems. Although Q&A forums serve as a good resource for seeking help from developers beyond the local team, the abundance of information can cause developers, especially novice software engineers, to spend considerable time in identifying relevant answers and suitable suggested fixes.
This exploratory study aims to understand how novice software engineers direct their efforts and what kinds of information they focus on within a post selected from the results returned in response to a search query on Stack Overflow. The results can be leveraged to improve the Q&A forum interface, guide tools for mining forums, and potentially improve granularity of traceability mappings involving forum posts. We qualitatively analyze the novice software engineers’ perceptions from a survey as well as their annotations of a set of Stack Overflow posts. Our results indicate that novice software engineers pay attention to only 27% of code and 15-21% of text in a Stack Overflow post to understand and determine how to apply the relevant information to their context. Our results also discern the kinds of information prominent in that focus.
Traditional approaches to the study of young children's behavior in helping relationships are examined and criticized as inadequate because they have failed to represent the child's perspective from the role of “active helpee” (i.e., help-seeker in such relationships). By failing to look at helping from the perspective of the one who seeks help, researchers have neglected to pursue an important lead in understanding why some children are able to learn and progress independently when confronted with the same obstacles that serve to defeat other children. This article focuses on instrumental help-seeking defined as an active, complex social-cognitive activity that is essential to learning and achievement. In the first sections of this article, it is argued that instrumental help-seeking can be formally distinguished from passive dependency as well as from the actual giving and receiving of help. In following sections, a heuristic model of the help-seeking process is offered, prior research relevant to the model is reviewed, and ideas for research on help-seeking in children are suggested within the framework of this model.
An Empirical Investigation of Collaborative Web Search Tool on Novice's Query Behavior
- Mareh Fakhir Al-Sammarraie
Carrie Grimes Diane Tang and Daniel Russell
- Carrie Grimes
- Diane Tang
- Daniel Russell
Otto Sepp"al"a, Petri Ihantola, Essi Isohanni, Juha Sorva, and Arto Vihavainen
- Otto Sepp
- Petri Ihantola
- Essi Isohanni
- Juha Sorva
- Arto Vihavainen
- Otto
Do we know how difficult the rainfall problem is
- Otto Sepp
- Petri Ihantola
- Essi Isohanni
- Otto