ArticlePublisher preview available

CROKAGE: effective solution recommendation for programming tasks by leveraging crowd knowledge

DOI:10.1007/s10664-020-09863-2
Authors:
Rodrigo F G Silva at Universidade Federal de Uberlândia (UFU)
Rodrigo F G Silva
Chanchal K. Roy at University of Saskatchewan
Chanchal K. Roy
Mohammad Masudur Rahman at Dalhousie University
Mohammad Masudur Rahman
Kevin A. Schneider
Kevin A. Schneider
Kevin A. Schneider
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Show all 7 authorsHide
Read publisher preview
Download citation
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

Developers often search for relevant code examples on the web for their programming tasks. Unfortunately, they face three major problems. First, they frequently need to read and analyse multiple results from the search engines to obtain a satisfactory solution. Second, the search is impaired due to a lexical gap between the query (task description) and the information associated with the solution (e.g., code example). Third, the retrieved solution may not be comprehensible, i.e., the code segment might miss a succinct explanation. To address these three problems, we propose CROKAGE (CrowdKnowledge Answer Generator), a tool that takes the description of a programming task (the query) as input and delivers a comprehensible solution for the task. Our solutions contain not only relevant code examples but also their succinct explanations written by human developers. The search for code examples is modeled as an Information Retrieval (IR) problem. We first leverage the crowd knowledge stored in Stack Overflow to retrieve the candidate answers against a programming task. For this, we use a fine-tuned IR technique, chosen after comparing 11 IR techniques in terms of performance. Then we use a multi-factor relevance mechanism to mitigate the lexical gap problem, and select the top quality answers related to the task. Finally, we perform natural language processing on the top quality answers and deliver the comprehensible solutions containing both code examples and code explanations unlike earlier studies. We evaluate and compare our approach against ten baselines, including the state-of-art. We show that CROKAGE outperforms the ten baselines in suggesting relevant solutions for 902 programming tasks (i.e., queries) of three popular programming languages: Java, Python and PHP. Furthermore, we use 24 programming tasks (queries) to evaluate our solutions with 29 developers and confirm that CROKAGE outperforms the state-of-art tool in terms of relevance of the suggested code examples, benefit of the code explanations and the overall solution quality (code + explanation).
Figures - available from: Empirical Software Engineering
This content is subject to copyright. Terms and conditions apply.
  • A preview of this full-text is provided by Springer Nature.
  • Learn more
Preview content only
Content available from Empirical Software Engineering
This content is subject to copyright. Terms and conditions apply.
https://doi.org/10.1007/s10664-020-09863-2
CROKAGE: effective solution recommendation
for programming tasks by leveraging
crowd knowledge
Rodrigo Fernandes Gomes da Silva1·Chanchal K. Roy2·
Mohammad Masudur Rahman2·Kevin A. Schneider2·Kl´
erisson Paix˜
ao1·
Carlos Eduardo de Carvalho Dantas1·Marcelo de Almeida Maia1
©Springer Science+Business Media, LLC, part of Springer Nature 2020
Abstract
Developers often search for relevant code examples on the web for their programming tasks.
Unfortunately, they face three major problems. First, they frequently need to read and anal-
yse multiple results from the search engines to obtain a satisfactory solution. Second, the
search is impaired due to a lexical gap between the query (task description) and the infor-
mation associated with the solution (e.g., code example). Third, the retrieved solution may
not be comprehensible, i.e., the code segment might miss a succinct explanation. To address
these three problems, we propose CROKAGE (CrowdKnowledge Answer Generator), a tool
that takes the description of a programming task (the query) as input and delivers a compre-
hensible solution for the task. Our solutions contain not only relevant code examples but also
their succinct explanations written by human developers. The search for code examples is
modeled as an Information Retrieval (IR) problem. We first leverage the crowd knowledge
stored in Stack Overflow to retrieve the candidate answers against a programming task. For
this, we use a fine-tuned IR technique, chosen after comparing 11 IR techniques in terms of
performance. Then we use a multi-factor relevance mechanism to mitigate the lexical gap
problem, and select the top quality answers related to the task. Finally, we perform natural
language processing on the top quality answers and deliver the comprehensible solutions
containing both code examples and code explanations unlike earlier studies. We evaluate
and compare our approach against ten baselines, including the state-of-art. We show that
CROKAGE outperforms the ten baselines in suggesting relevant solutions for 902 pro-
gramming tasks (i.e., queries) of three popular programming languages: Java, Python and
PHP. Furthermore, we use 24 programming tasks (queries) to evaluate our solutions with
29 developers and confirm that CROKAGE outperforms the state-of-art tool in terms of
relevance of the suggested code examples, benefit of the code explanations and the overall
solution quality (code + explanation).
Communicated by: Tim Menzies
Marcelo de Almeida Maia
marcelo.maia@ufu.br
Extended author information available on the last page of the article.
Empirical Software Engineering (2020) 25:4707–4758
Published online: 2 September 2020
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
... In this step, we selected 10,000 input queries performed by users on CROKAGE tool 3 . CROKAGE is a code search engine that extract code snippets written in Java language and their explanations from StackOverflow [4]. These input queries were performed by users from more than 80 countries, searching for programming tasks 4 . ...
... CROKAGE is a code search engine that extract code snippets written in Java language and their explanations from StackOverflow [4]. These input queries were performed by users from more than 80 countries, searching for programming tasks 4 . We removed duplicated queries and queries manually labeled as not applicable (e.g., non-Java programming languages) by the CROKAGE research [4]. ...
... These input queries were performed by users from more than 80 countries, searching for programming tasks 4 . We removed duplicated queries and queries manually labeled as not applicable (e.g., non-Java programming languages) by the CROKAGE research [4]. ...
Readability and Understandability of Snippets Recommended by General-purpose Web Search Engines: a Comparative Study
Preprint
  • Oct 2021
Developers often search for reusable code snippets on general-purpose web search engines like Google, Yahoo! or Microsoft Bing. But some of these code snippets may have poor quality in terms of readability or understandability. In this paper, we propose an empirical analysis to analyze the readability and understandability score from snippets extracted from the web using three independent variables: ranking, general-purpose web search engine, and recommended site. We collected the top-5 recommended sites and their respective code snippet recommendations using Google, Yahoo!, and Bing for 9,480 queries, and evaluate their readability and understandability scores. We found that some recommended sites have significantly better readability and understandability scores than others. The better-ranked code snippet is not necessarily more readable or understandable than a lower-ranked code snippet for all general-purpose web search engines. Moreover, considering the readability score, Google has better-ranked code snippets compared to Yahoo! or Microsoft Bing
... A web crawler is a program that requests websites and extracts information from them [30]. In the study of music recommendation systems, there are numerous data sets. ...
Intelligent Recommendation Model of Contemporary Pop Music Based on Knowledge Map
Article
Full-text available
With the advent of the era of big data, the rise of Web2.0 completely subverts the traditional Internet model and becomes the trend of today’s information age. Simultaneously, massive amounts of data and information have infiltrated various Internet companies, resulting in an increase in the problem of information overload. In the online world, learning how to quickly and accurately select the parts we are interested in from a variety of data has become a hot topic. Intelligent music recommendation has become a current research hotspot in music services as a viable solution to the problem of information overload in the digital music field. On the basis of precedents, this paper examines the characteristics of music in a comprehensive and detailed manner. A knowledge graph-based intelligent recommendation algorithm for contemporary popular music is proposed. User-defined tags are described as the free genes of music in this paper, making it easier to analyze user behavior and tap into user interests. It has been confirmed that this algorithm’s recommendation quality is relatively high, and it offers a new development path for improving the speed of searching for health information services.
... A motivating example of subjective perceptions is shown in Figure 1. This example has the first code snippet suggested by Google, Microsoft Bing and CROKAGE 2 (tool that provides code snippets and their correspond comprehensive solution for each input query, both mined from Stack Overflow [5]) for the input query Find maximum element of ArrayList in Java. The Table 1 shows the readability [21] and understandability [4] score for each suggested code snippet. ...
Readability and Understandability Scores for Snippet Assessment: an Exploratory Study
Preprint
Full-text available
  • Aug 2021
Code search engines usually use readability feature to rank code snippets. There are several metrics to calculate this feature, but developers may have different perceptions about readability. Correlation between readability and understandability features has already been proposed, i.e., developers need to read and comprehend the code snippet syntax, but also understand the semantics. This work investigate scores for understandability and readability features, under the perspective of the possible subjective perception of code snippet comprehension. We find that code snippets with higher readability score has better comprehension than lower ones. The understandability score presents better comprehension in specific situations, e.g. nested loops or if-else chains. The developers also mentioned writability aspects as the principal characteristic to evaluate code snippets comprehension. These results provide insights for future works in code comprehension score optimization.
... Stack Overflow (SO) is the most prominent example of such service, serving more than 100M users monthly, with more than 20M registered questions 1 . The crowd knowledge available on Stack Overflow dumps have enabled several studies that leverage such raw content to produce documentation for APIs [1], recommend posts [2,3] and answers from queries [4,5], understand social interactions [6], and several others 2 . ...
Towards a question answering assistant for software development using a transformer-based language model
Preprint
Full-text available
  • Mar 2021
Question answering platforms, such as Stack Overflow, have impacted substantially how developers search for solutions for their programming problems. The crowd knowledge content available from such platforms has also been used to leverage software development tools. The recent advances on Natural Language Processing, specifically on more powerful language models, have demonstrated ability to enhance text understanding and generation. In this context, we aim at investigating the factors that can influence on the application of such models for understanding source code related data and produce more interactive and intelligent assistants for software development. In this preliminary study, we particularly investigate if a how-to question filter and the level of context in the question may impact the results of a question answering transformer-based model. We suggest that fine-tuning models with corpus based on how-to questions can impact positively in the model and more contextualized questions also induce more objective answers.
Improved retrieval of programming solutions with code examples using a multi-featured score
Article
Full-text available
Developers often depend on code search engines to obtain solutions for their programming tasks. However, finding an expected solution containing code examples along with their explanations is challenging due to several issues. There is a vocabulary mismatch between the search keywords (the query) and the appropriate solutions. Semantic gap may increase for similar bag of words due to antonyms and negation. Moreover, documents retrieved by search engines might not contain solutions containing both code examples and their explanations. So, we propose CRAR (Crowd Answer Recommender) to circumvent those issues aiming at improving retrieval of relevant answers from Stack Overflow containing not only the expected code examples for the given task but also their explanations. Given a programming task, we investigate the effectiveness of combining information retrieval techniques along with a set of features to enhance the ranking of important threads (i.e., the units containing questions along with their answers) for the given task and then selects relevant answers contained in those threads, including semantic features, like word embeddings and sentence embeddings, for instance, a Convolutional Neural Network (CNN). CRAR also leverages social aspects of Stack Overflow discussions like popularity to select relevant answers for the tasks. Our experimental evaluation shows that the combination of the different features performs better than each one individually. We also compare the retrieval performance with the state-of-art CROKAGE (Crowd Knowledge Answer Generator), which is also a system aimed at retrieving relevant answers from Stack Overflow. We show that CRAR outperforms CROKAGE in Mean Reciprocal Rank and Mean Recall with small and medium effect sizes, respectively.
Towards a question answering assistant for software development using a transformer-based language model
Conference Paper
  • Jun 2021
Duplicate Question Detection in Stack Overflow: A Reproducibility Study
Preprint
Full-text available
  • Feb 2018
Stack Overflow has become a fundamental element of developer toolset. Such influence increase has been accompanied by an effort from Stack Overflow community to keep the quality of its content. One of the problems which jeopardizes that quality is the continuous growth of duplicated questions. To solve this problem, prior works focused on automatically detecting duplicated questions. Two important solutions are DupPredictor and Dupe. Despite reporting significant results, both works do not provide their implementations publicly available, hindering subsequent works in scientific literature which rely on them. We executed an empirical study as a reproduction of DupPredictor and Dupe. Our results, not robust when attempted with different set of tools and data sets, show that the barriers to reproduce these approaches are high. Furthermore, when applied to more recent data, we observe a performance decay of our both reproductions in terms of recall-rate over time, as the number of questions increases. Our findings suggest that the subsequent works concerning detection of duplicated questions in Question and Answer communities require more investigation to assert their findings.
Recommending Comprehensive Solutions for Programming Tasks by Mining Crowd Knowledge
Conference Paper
Full-text available
  • May 2019
Prediction of relatedness in stack overflow: deep learning vs. SVM: a reproducibility study
Conference Paper
Full-text available
  • Oct 2018
Background Xu et al. used a deep neural network (DNN) technique to classify the degree of relatedness between two knowledge units (question-answer threads) on Stack Overflow. More recently, extending Xu et al.'s work, Fu and Menzies proposed a simpler classification technique based on a fine-tuned support vector machine (SVM) that achieves similar performance but in a much shorter time. Thus, they suggested that researchers need to compare their sophisticated methods against simpler alternatives. Aim The aim of this work is to replicate the previous studies and further investigate the validity of Fu and Menzies' claim by evaluating the DNN- and SVM-based approaches on a larger dataset. We also compare the effectiveness of these two approaches against SimBow, a lightweight SVM-based method that was previously used for general community question-answering. Method We (1) collect a large dataset containing knowledge units from Stack Overflow, (2) show the value of the new dataset addressing shortcomings of the original one, (3) re-evaluate both the DNN-and SVM-based approaches on the new dataset, and (4) compare the performance of the two approaches against that of SimBow. Results We find that: (1) there are several limitations in the original dataset used in the previous studies, (2) effectiveness of both Xu et al.'s and Fu and Menzies' approaches (as measured using F1-score) drop sharply on the new dataset, (3) similar to the previous finding, performance of SVM-based approaches (Fu and Menzies' approach and SimBow) are slightly better than the DNN-based approach, (4) contrary to the previous findings, Fu and Menzies' approach runs much slower than DNN-based approach on the larger dataset - its runtime grows sharply with increase in dataset size, and (5) SimBow outperforms both Xu et al. and Fu and Menzies' approaches in terms of runtime. Conclusion We conclude that, for this task, simpler approaches based on SVM performs adequately well. We also illustrate the challenges brought by the increased size of the dataset and show the benefit of a lightweight SVM-based approach for this task.
Detecting Misflagged Duplicate Questions in Community Question-Answering Archives
Conference Paper
Full-text available
  • Jun 2018
In this paper we introduce the task of misflagged duplicate question detection for question pairs in community question-answer (cQA) archives and compare it to the more standard task of detecting valid duplicate questions. A misflagged duplicate is a question that has been erroneously hand-flagged by the community as a duplicate of an archived one, where the two questions are not actually the same. We find that for misflagged duplicate detection, meta data features that capture user authority, question quality, and relational data between questions, outperform pure text-based methods, while for regular duplicate detection a combination of meta data features and semantic features gives the best results. We show that misflagged duplicate questions are even more challenging to model than regular duplicate question detection, but that good results can still be obtained.
Deep code comment generation
Conference Paper
Full-text available
  • May 2018
During software maintenance, code comments help developers comprehend programs and reduce additional time spent on reading and navigating source code. Unfortunately, these comments are often mismatched, missing or outdated in the software projects. Developers have to infer the functionality from the source code. This paper proposes a new approach named DeepCom to automatically generate code comments for Java methods. The generated comments aim to help developers understand the functionality of Java methods. DeepCom applies Natural Language Processing (NLP) techniques to learn from a large code corpus and generates comments from learned features. We use a deep neural network that analyzes structural information of Java methods for better comments generation. We conduct experiments on a large-scale Java corpus built from 9,714 open source projects from GitHub. We evaluate the experimental results on a machine translation metric. Experimental results demonstrate that our method DeepCom outperforms the state-of-the-art by a substantial margin.
Deep code search
Conference Paper
Full-text available
  • May 2018
To implement a program functionality, developers can reuse previously written code snippets by searching through a large-scale codebase. Over the years, many code search tools have been proposed to help developers. The existing approaches often treat source code as textual documents and utilize information retrieval models to retrieve relevant code snippets that match a given query. These approaches mainly rely on the textual similarity between source code and natural language query. They lack a deep understanding of the semantics of queries and source code. In this paper, we propose a novel deep neural network named CODEnn (Code-Description Embedding Neural Network). Instead of matching text similarity, CODEnn jointly embeds code snippets and natural language descriptions into a high-dimensional vector space, in such a way that code snippet and its corresponding description have similar vectors. Using the unified vector representation, code snippets related to a natural language query can be retrieved according to their vectors. Semantically related words can also be recognized and irrelevant/noisy keywords in queries can be handled. As a proof-of-concept application, we implement a code search tool named DeepCS using the proposed CODEnn model. We empirically evaluate DeepCS on a large scale codebase collected from GitHub. The experimental results show that our approach can effectively retrieve relevant code snippets and outperforms previous techniques.
Effective Reformulation of Query for Code Search Using Crowdsourced Knowledge and Extra-Large Data Analytics
Conference Paper
Full-text available
  • Jun 2018
Software developers frequently issue generic natural language queries for code search while using code search engines (e.g., GitHub native search, Krugle). Such queries often do not lead to any relevant results due to vocabulary mismatch problems. In this paper, we propose a novel technique that automatically identifies relevant and specific API classes from Stack Overflow Q & A site for a programming task written as a natural language query, and then reformulates the query for improved code search. We first collect candidate API classes from Stack Overflow using pseudo-relevance feedback and two term weighting algorithms, and then rank the candidates using Borda count and semantic proximity between query keywords and the API classes. The semantic proximity has been determined by an analysis of 1.3 million questions and answers of Stack Overflow. Experiments using 310 code search queries report that our technique suggests relevant API classes with 48% precision and 58% recall which are 32% and 48% higher respectively than those of the state-of-the-art. Comparisons with two state-of-the-art studies and three popular search engines (e.g., Google, Stack Overflow, and GitHub native search) report that our reformulated queries (1) outperform the queries of the state-of-the-art, and (2) significantly improve the code search results provided by these contemporary search engines.
Mining Likely Analogical APIs across Third-Party Libraries via Large-Scale Unsupervised API Semantics Embedding
Article
Establishing API mappings between third-party libraries is a prerequisite step for library migration tasks. Manually establishing API mappings is tedious due to the large number of APIs to be examined. Having an automatic technique to create a database of likely API mappings can significantly ease the task. Unfortunately, existing techniques either adopt supervised learning mechanism that requires already-ported or functionality similar applications across major programming languages or platforms, which are difficult to come by for an arbitrary pair of third-party libraries, or cannot deal with lexical gap in the API descriptions of different libraries. To overcome these limitations, we present an unsupervised deep learning based approach to embed both API usage semantics and API description (name and document) semantics into vector space for inferring likely analogical API mappings between libraries. Based on deep learning models trained using tens of millions of API call sequences, method names and comments of 2.8 millions of methods from 135,127 GitHub projects, our approach significantly outperforms other deep learning or traditional information retrieval (IR) methods for inferring likely analogical APIs. We implement a proof-of-concept website which can recommend analogical APIs for 583,501 APIs of 111 pairs of analogical Java libraries with diverse functionalities. This scale of third-party analogical-API database has never been achieved before.
API method recommendation without worrying about the task-API knowledge gap
Conference Paper
  • Sep 2018
Developers often need to search for appropriate APIs for their programming tasks. Although most libraries have API reference documentation, it is not easy to find appropriate APIs due to the lexical gap and knowledge gap between the natural language description of the programming task and the API description in API documentation. Here, the lexical gap refers to the fact that the same semantic meaning can be expressed by different words, and the knowledge gap refers to the fact that API documentation mainly describes API functionality and structure but lacks other types of information like concepts and purposes, which are usually the key information in the task description. In this paper, we propose an API recommendation approach named BIKER (Bi-Information source based KnowledgE Recommendation) to tackle these two gaps. To bridge the lexical gap, BIKER uses word embedding technique to calculate the similarity score between two text descriptions. Inspired by our survey findings that developers incorporate Stack Overflow posts and API documentation for bridging the knowledge gap, BIKER leverages Stack Overflow posts to extract candidate APIs for a program task, and ranks candidate APIs by considering the query’s similarity with both Stack Overflow posts and API documentation. It also summarizes supplementary information (e.g., API description, code examples in Stack Overflow posts) for each API to help developers select the APIs that are most relevant to their tasks. Our evaluation with 413 API-related questions confirms the effectiveness of BIKER for both class- and method-level API recommendation, compared with state-of-the-art baselines. Our user study with 28 Java developers further demonstrates the practicality of BIKER for API search.
Learning to mine aligned code and natural language pairs from stack overflow
Conference Paper
  • May 2018
For tasks like code synthesis from natural language, code retrieval, and code summarization, data-driven models have shown great promise. However, creating these models require parallel data between natural language (NL) and code with fine-grained alignments. Stack Overflow (SO) is a promising source to create such a data set: the questions are diverse and most of them have corresponding answers with high quality code snippets. However, existing heuristic methods (e.g., pairing the title of a post with the code in the accepted answer) are limited both in their coverage and the correctness of the NL-code pairs obtained. In this paper, we propose a novel method to mine high-quality aligned data from SO using two sets of features: hand-crafted features considering the structure of the extracted snippets, and correspondence features obtained by training a probabilistic model to capture the correlation between NL and code using neural networks. These features are fed into a classifier that determines the quality of mined NL-code pairs. Experiments using Python and Java as test beds show that the proposed method greatly expands coverage and accuracy over existing mining methods, even when using only a small number of labeled examples. Further, we find that reasonable results are achieved even when training the classifier on one language and testing on another, showing promise for scaling NL-code mining to a wide variety of programming languages beyond those for which we are able to annotate data.