Conference PaperPDF Available

Static analysis of source code written by novice programmers

Authors:

Figures

Content may be subject to copyright.
Static analysis of source code written by novice
programmers
Abstract In this paper we are reporting the finding on the use
of a static analysis of C source code written by students learning
to program. Two different tools for static code analysis were used
to analyze the solutions submitted by the students on the partial
exams and exams from the introductory course in programming
in a three year period. We have collected, analyzed and compared
most common errors reported by both tools. We further
investigate if the available checks provided by these tools, often
used in professional software development practices to find bugs
and improve the code quality, can also help novice programmers
in tracking down and resolving their problems in the code or have
any other value in the process of learning programming.
Keywordsstatic analysis; novice programmers; programming
errors;
I. INTRODUCTION
The term static analysis refers to any process of assessing
source code without executing it. It is often used by experienced
programmers to find bugs, memory leaks or other potential
problems in programs. The information reported from static
analysis is used to improve code quality, security, robustness,
and in some occasions, cyclomatic complexity. Example of
errors and problems discovered with static analysis are
uninitialized variables, unreachable code, resource leaks, unused
variables and many others. Static analysis is also used in
dynamically typed programming languages to check for coding
standard rules and type errors. In some cases errors discovered
with static analysis are indication of serious flaws in program
logic and can be potential cause for bugs. For this reason in many
complex software projects, the static analysis is automated by
including it in the build process. The programmers on the project
can use the reports from this analysis to find and fix bugs or
improve the code quality. They can also use it to learn about
hidden features of the programming language, leading to bugs
or unusual behavior of the program [1].
All syllabi for CS majors as well as most syllabi for
engineering and science majors do include a course for
programming in the introductory year of university studies.
Although the freshman now days are much more exposed to
computers and technology during high school (secondary
education) and some of them have even taken programming
courses in high school, programming has not become easier
subject. Most researchers agree that in order to master
programming one needs some theory and a lot of practice
(learning by doing). That is why most programming courses are
organized as formal lectures plus a significant number of classes
where the students are challenged to solve problems by
themselves, usually denoted as laboratory exercises. From the
students’ point of view this is a form of deliberate practice that
is not just simple repetition of the exercise, but challenging
oneself with more involving tasks. These tasks should push
students beyond their current abilities, and while working on
them, they should analyze their performance during and after,
and learn and correct the mistakes they are making.
Introductory courses in programming are usually enrolled by
large number of students and most of them are with very limited
or no programming experience at all. We refer to the students
that might have some limited previous experience, or have never
before studied or practiced programming as novice
programmers. Having different background knowledge and
aptitude for programming, novice programmers have
documented difficulties in learning and understanding
programming. These difficulties and misconceptions are often
manifested when they try to write programs as part of the formal
assessment and examination. Analyzing the source code written
by novice programmers can help in discovering the most
common errors they make, concepts they find difficult or
misconceptions they form.
In this paper, we are exploring the possibility of introducing
and applying static analysis on source code written by novice
programmers. The goal of this work is to identify the most
common error they make and investigate if using tools for static
analysis can be valuable to them. We applied two tools for static
analysis on dataset of solutions written by novice programmers
as part of their exams in a three year period (2013-2015).
The rest of the paper is organized as follows. In section II we
present the common challenges of novice programmers when
leaning programming and the related research in this area. The
used tools for static analysis are described in section III and in
section IV, the methodology of the conducted experiment is
explained. Finally the experimental results are presented and
discussed in section V, and in the last section, a short summary
and conclusion is given.
II. LEARNING PROGRAMMING CHALLENGES BY NOVICE
PROGRAMMERS
Programming is an individual skill acquired through practice
and experience over time. The novice learner, in order to learn
to program, must learn the syntax of a programming language
and master many skills and concepts, such as constructing a
mental model of a notional machine [2]. Novices start writing
programs with a very little idea of the properties of the notional
machine implied by the constructs of the programming language
they are learning [3]. There are many documented cases where
novices form faulty mental model of the program dynamics
originated from their limited pre-programming knowledge [4].
These limitations can cause difficulties in constructing correct
programs and solving problems. For example they can have
difficulties in perceiving a piece of code as isolated component
rather than an active component of dynamic process that occurs
at runtime [5].
To apply static analysis on source code written from novice
programmers, first we should introduce the profile of the novice
programmer along with their mental model. The novice
programmers face many well documented learning challenges in
the process of learning programming [6, 7] that we should
explore before applying or introducing any new technique in
their learning process.
By studying the difficulties, misconceptions or common
errors the novice programmers are making we can achieve better
understanding of the problem-solving strategies and highlight
the difficult aspects of programming. Furthermore, such studies
can spark contributions in refining of existing or invention of
new programming languages, training tools and teaching
methods. One approach on identifying the misconceptions of
programming is by interviewing students using think-aloud
protocol on closed list of problems covering different concepts
such as control flow, types, conditionals and others [8]. Other
approach can be by polling the students with questionnaire on
different topics and concepts. For example one such study [7]
shows that topics that rely on clear understanding of pointers and
memory-related concepts proved to be the most difficult. The
approach we have taken in this paper was to collect and process
the reports of applying static analysis on the code written by
novices, and then quantify and analyze the reported errors. The
analyzed source code files are solutions submitted by novice
programmers on problems given on partial exams and exams.
For many novice programmers, the first interaction with the
tools needed to compile, run and test their program is
challenging. To address this issue, in the past years in the field
of teaching and learning programming many automated
programming assessment systems are gaining popularity [9].
Such systems are considered to significantly lower the entry
barrier for novice programmers caused by complicated tools or
integrated development environments (IDE's). Our solution to
these problems, is a system named Code [10], developed at the
Faculty of Computer Science and Engineering at the Ss. Cyril
and Methodius University in Skopje, used for automatic
assessment and management of students' programming
exercises and exams. The system provides a simple web-based
user interface, where the students can write, run and test their
solutions (fig. 1). The solutions are automatically stored,
compiled and executed on a central server. One of the main
advantages of the system is the automatic assessment with
immediate feedback, which can help both, the course instructors
and the students. Providing timely and informative feedback in
such systems is important component [11] especially in
situations when direct feedback from instructors is not feasible.
The feedback can vary from plain compilation output, to hints
of errors in the program, or listing and comparing the test input
data and the expected output. The type of feedback provided is
important and can affect the strategy imposed in the process of
constructing a working solution.
One idea investigated in this work is the possibility of using
static analysis tools to enrich the feedback with valuable
information for potential bugs and errors. Static analysis can be
used to check programming style, program errors (syntax or
semantic), software metrics assessment, structural and non-
structural similarity analysis, keywords analysis, plagiarism
detection and diagram assessment [12]. It has also been
successfully used to help students writing better code [13].
Fig. 1. Web-based interface of the automatic assessment system Code
III. STATIC ANALYSIS TOOLS
Static analysis is performed with specialized tools
developed only for that purpose. A static analysis tool can
explore large number of “what if” scenarios without having to
go through all the computations necessary to execute the code
for all scenarios. Also, good static analysis tools provide a fast
way to create complete and consistent evaluation report of the
source code. Most often these tools are specialized for
evaluation of source files written in single programming
language. However, there are some more general tools capable
of evaluating source files in multiple, often similar programming
languages.
The source code of the solutions analyzed in this study was
written in the C programming language. We have tried the
following tools for static analysis of C code: Clang Static
Analyzer, CppCheck, Splint, OClint and the first two were
chosen for this analysis. Our motivation was to select the most
appropriate ones for our context of introduction level of
programming and the programming language C, and not to
review tools for static analysis of C source code. The available
checks and the reporting of errors these tools can provide, also
was an important factor in this decision. Clang Static Analyzer
and CppCheck were chosen as they were found to provide
distinct types of checks and warnings about potential source
code problems. The first one also includes some recently
introduced experimental checkers relevant to our research, and
also it's used as underlying checker of one of the remaining static
analysis tools OClint.
A. Clang Static Analyzer
The Clang Static Analyzer [14] is a source code analysis tool
part of Clang which is a C, C++ and Objective-C front-end of
the LLVM compiler [15]. The Clang Static Analyzer can spot
errors and bugs with different degrees of analysis sophistication,
mostly using the control-flow graph of the program. Finding
errors and potential bugs is implemented by various checkers
that are grouped in the following six groups: core checkers, C++
checkers, dead code checkers, OS X checkers, security checkers
and UNIX checkers.
In our analysis we used the core checkers which model core
language features and perform general-purpose checks such as
division by zero, null pointer dereference, usage of uninitialized
values and errors. We also used dead code checkers and some of
the experimental checkers. Example of some of these checks are
shown in table 1.
B. CppCheck
CppCheck is a static analysis tool for C/C++ code which
primarily detects the types of bugs that the compilers normally
do not detect, such as: out of bounds checking, memory leaks
checking, detect possible null pointer dereferences, check for
uninitialized variables, and few others. CppCheck can also be
extended with simple patterns, defining rules for functions or
scripts.
TABLE I. EXAMPLE CHECKS FROM CLANG STATIC ANALYZER
Name, Description
Example
core.CallAndMessage (C, C++,
ObjC)
Check for logical errors for
function calls (e.g., uninitialized
arguments, null function pointers).
void f(int x);
void test() {
int x;
f(x); // warn: passed-
by-value // arg contain
uninitialized data
}
alpha.deadcode.UnreachableCode
(C, C++, ObjC)
Check unreachable code.
int test() {
int x = 1;
while(x);
return x; // warn
}
IV. METHODOLOGY
The study is based on the data collected during an
introductory programming course “Structured programming”
using the C programming language. The enrollment (including
re-enrollment) on this course reaches up to 1000 students. The
course covers the basic programming constructs and structured
programming concepts in 14 weeks with 2 forty-five minutes
long lectures held by professors and 2 equally long problems
solving sessions given by teaching assistants. The concepts
covered are reading and writing to I/O, flow control, iteration
and recursion, arrays (two-dimensional), pointers and reading or
writing files. Most of the example problems can be solved by
constructing a short and simple straightforward algorithm.
Students also participate in 10 (hour and a half long) hands-
on laboratory sessions where they are presented with several
exercise problems that they supposed to solve individually. The
assessment is organized in two partial exams and a final exam
(during the semester), followed by two additional final exams
sessions during the academic year. The exams usually include
up to four problems, on which the students are expected to write
complete working solutions. Exams are taken in controlled
laboratory environment, where students can write, run and test
their solutions in the web-based environment for programming
and automatic assessment Code. When they open a problem in
the system Code, they have a side by side view of the problem
text and a web-based text editor (fig. 1), where they can write
their solution. Students also can use other editors or IDE’s to
compile and execute their solutions on the local machine, but are
required to post the final solution in the system. Once a student
has written a solution, he/she can make one of two possible
actions, run or submit. On the action run, the solution is first
compiled and then if the compilation is successful, it’s executed
with a sample test input which is visible in the problem view. In
case of compilation errors the output of the compiler is
presented, and otherwise the output of the execution is presented
next to the expected output for the given test input. The students
can visually compare both, the output from the execution of their
program and the given expected output. On the submit action,
the solution is compiled and then executed with multiple test
cases as input. For each test case the output of the execution of
the students solution is compared to the expected output for that
test case, and a report of the comparison is presented to the
student. Some of the test cases can be hidden (the input, the
expected and actual output are not shown) and only the result
(passed/failed) is reported. Students can make unlimited number
of submit attempts.
On each run or submit attempt the system automatically
provides three types of feedback. The first type of feedback is
the output from the compilation process before execution with
the standard report from the GCC or Clang compiler (fig. 2).
Fig. 2. Compilation output feedback
The second type of feedback is the report from dynamic
analysis of their solutions (fig. 3). The dynamic analysis is
testing the correctness of the solution by comparing the output
from the execution with a predefined test cases for the problem.
Also, as third type of feedback an HTML report from analyzing
the source code with the Clang Static Analyzer is available.
Fig. 3. Dynamic analysis feedback for solution.
In this study we have assembled a dataset containing
students’ solutions on the problems given on the exams in the
academic years 2013, 2014 and 2015. To build the dataset, a
total of 13,960 source code files submitted as final solutions to
the exam problems were extracted from the system Code. We
used the selected tools for static analysis on the full dataset, and
analyzed the generated reports. The Clang Static Analyzer can
only be performed on source code files that can be compiled
successfully, while all source code files (including those that
cannot be compiled) can be processed with CppCheck. Both
tools can be used as command-line programs that can be
executed on a group of files and they report the results in semi-
structured format. The Clang Static Analyzer also outputs a
visual report in HTML file, where the found errors are described
in more details. The data used for reports was extracted by
processing and parsing the reports generated by using the static
analysis tools on the dataset.
From our observations on a sample of the results from the
exams, we can identify four groups of students by their
performance. The first and usually small group of talented
students, with almost perfect score, are the ones who solve the
exam problems without any obstacles. The second group,
students who passed the exam with solving at least one problem
without mistakes, consist of students who can solve most of the
problems with some obstacles, but might make small mistakes
on the path. The third group of students, is identified as the group
that can write some working code for simple assignments, but
have problems to create or deduce a clear mental model for more
complicated programming constructs such as arrays, nested
loops or recursion. The fourth and last group, cannot write any
working code and in most cases is unmotivated to perform.
V. EXPERIMENTAL RESULTS
A. Compilation results
As a first step of the analysis, compilation was attempted on
each solution of the dataset. Here we report the results of
compilation success rate for each exam. Only successfully
compiled solutions were used in the further analysis with the
Clang Static Analyzer. The results showed that almost 25% of
the source files contained at least one compilation error.
Significant percentage of them were empty or contained some
free text (detailed results are shown in table 2). Errors in
compilation are indication of lack of knowledge of the syntax of
the C programming language. Also some students have
manifested difficulties in understanding and then acting upon the
compiler messages. There are also few cases when students are
not motivated or interested in writing solution of the problem, so
they write random content that does not compile.
TABLE II. PERCENTAGE OF SUCCESSFULLY COMPILED SOLUTIONS
Year
Partial
exam 1
Partial
exam 2
January
exam
August
exam
2013
74.1%
81.7%
73.4%
73.3%
2014
74.8%
81.6%
71.8%
73.3%
2015
79.1%
87.0%
64.8%
75.8%
On figure 3 the overall compilation success over the time of
one year is shown, starting from the first partial exam in
November and ending with the last exam in August. The highest
rate in compilation on the second partial exam, can be explained
by the fact that only students that scored over 40% on the first
partial exam are allowed to take the second. These are usually
students that have already proven their basic knowledge in
programming and the programming language syntax. The final
exam is only taken by the students that have scored less than the
passing minimum of the partial exams. Also, noticeable is the
consistency in the change of the compilation success rate on
same exams across the analyzed period of three years.
Fig. 4. Compilation success over academic year
B. Clang Static Analysis report results
Running Clang Static Analysis tool on the dataset, produced
detailed report with all the detected errors for each source file
containing errors. The errors are grouped by two types of checks,
denoted as Dead store and Logic errors.
Dead store check includes type of checks such as dead
assignment, dead initializations or dead increment. Dead
assignment (fig. 5) is a common error where a variable is
assigned some value, and later in the code the assigned variable
is never used. Dead initialization is a similar error where variable
is initialized with some value, and in the rest of the code that
variable is never used. Dead increment is a type of error where
a variable is mutated (incremented or decremented), and then
this new value of the variable is never used. The distribution of
these errors it’s shown on figure 6. These kind of errors such as
incorrect variable declaration and usage are very common
between novice programmers and in many cases are serious
indication of a wrong solution. Early reporting on such errors
can be helpful in finding a potential bug caused by this kind of
errors. From all the errors reported, 34% turned out to be dead
store errors.
Fig. 5. Dead assignment error
Fig. 6. Dead store errors distribution
Dead code is experimental check for unreachable code,
which is a block of code that will never be executed, often as a
consequence of previous endless loop or incorrect branching
condition. This type of error is difficult to find and determine in
compilation time using only static analysis. For example using
the function scanf for reading from standard input can be a
source for false positives or missed endless loops. This error
(when found successfully) is a clear indication of wrong
solution, or wrong implementation of an algorithm. This can be
very useful information for novice programmers, because it is
type of error not commonly found by compilers. To be able to
find these kind of errors, which are usually found at runtime,
requires very good understanding of the program dynamics and
execution flow. Example of reported dead code error is shown
on figure 7.
Fig. 7. Dead code error
Logic errors are the most common type of errors made by
novice programmers. As shown on table 3, 66% of all errors are
logic errors. Of all logic errors detected, following four types are
the most common: result of operation is garbage or undefined,
uninitialized argument value, assigned value is garbage or
undefined and out-of-bound access. The precise distribution can
be seen on figure 9, where the fifth group denoted as other
includes all other reported errors. Errors such as these are related
with concepts such as conditional logic, arithmetic operations,
and variable initializations and updating. The Clang Static
Analyzer does a very good job in tracking the logic errors
through the code, and traces the steps that have caused the error
(as shown on figure 8). The report produced by this analyzer can
be especially helpful to novices not just for locating the error,
but also for learning and understanding how it happened in the
first place.
Fig. 8. Logic errors
The total number of errors and percentages of source files
with at least one error found with static analysis using the Clang
Static Analyzer are presented on table 3. Significant percentage
of the dead store error were false positives, caused by the above
mentioned problem with the scanf function. Still, the large
numbers are showing that using this tool can be helpful for
students to find many types of errors, they often fail to notice
with standard compiler or just by running and inspecting the
code.
TABLE III. CLANG STATIC ANALYZER ERRORS REPORT
Year
Dead store
errors
Logic
errors
Total solutions
analyzed
% of solutions
with at least one
error
2013
784
2,369
3,333
54.06%
2014
827
2,646
3,552
57.48%
2015
1,001
2,675
3,477
61.75%
Total
2,612 (34%)
7,690 (66%)
10,362
57.56%
Fig. 9. Logic errors distribution
C. CppCheck results
CppCheck static analysis tool produces reports on 5 groups
of errors: performance, style, warning, portability and error. In
our study we ignored the performance and portability checks as
not relevant for the novice programmers and focused only on the
other three types. In table 4 the most common types of checks
reported by CppCheck are summarized.
TABLE IV. TYPES OF CHECKS FROM CPPCHECK
Type
Example
Style
Unused variables
The scope of variable ‘x’ can be reduced
Variable ‘x’ is assigned a value that is never used
Variable ‘x’ is not assigned a value
Warning
%d in format string (no. 1) requires 'int *' but the
argument type is 'int'
printf format string requires 0 parameters but 1 is
given
%d in format string (no. 1) requires 'int' but the
argument type is 'int *'
String literal compared with variable 'n'. Did you
intend to use strcmp() instead?
Comparison of a boolean expression with an integer
other than 0 or 1
Error
Uninitialized variable
Invalid number of character ({) when these macros
are defined
Array index -1 is out of bounds
The errors reported from this tool includes unused variables,
wrong comparison expressions, array indexes out of bounds and
others that are common between novice programmers. The
results on table 5 are showing that novice programmers are
making large number of these errors, with an average of more
than 2 (style/warning/error) per solution.
TABLE V. RESULTS FROM STATIC ANALYSIS USING CPPCHECK
Year
Total solutions
Style
Warning
Error
2013
4,491
5,512
1,772
2,183
2014
4,815
6,435
1,139
2,609
2015
4,654
6,381
1,394
2,645
VI. DISCUSSION AND FUTURE WORK
Debugging and finding errors is proven to be very difficult
task, which is particularly difficult and frustrating for novice
programmers. Students have indicated that the easiest bugs to fix
are those found by the compiler or some other tool. This suggests
that for students, locating bugs is more difficult than fixing them,
or that the types of bugs found by the compiler and other tools,
which are most often construct-related, are easier to fix than
other bugs [16]. The results reported in this paper, are
confirming these findings, by showing that tools for static
analysis can be helpful to novice students in finding and
understanding bugs. Incorporating such tool in the students’
learning environment for programming such as Code, can bring
multiple advantages. It can help students to find logic errors,
learn about them and consequently make them better in the
debugging process.
Static analysis is proven and widely used tool by
professionals to find bugs and code smells in large code bases.
It is mostly used to help avoiding bed practices or find and track
down bugs. In this study we have used two industry tools for
static analysis to process the source code written from novice
programmers during exam sessions. The results are confirming
that novice programmers make large number of mistakes such
as “uninitialized variable”, or styling types of error such as
“unused variable”. Most of these errors are left unnoticed by
novices, and are often cause for wrong solutions, even when
their initial idea for the algorithm is correct. This study shows
that including static analysis in the learning process can have
value, mostly because it can help novices in finding individually
many common errors they make. How will they actually accept
and act on the reports on the errors can be an interesting question
we can try to answer in some future work.
ACKNOWLEDGMENT
The work presented in this paper was partially financed by the
Faculty of Computer Science and Engineering at the Ss. Cyril
and Methodius University in Skopje, Macedonia.
REFERENCES
[1] Emanuelsson, P., Nilsson, U.: A comparative study of industrial static
analysis tools. Electronic notes in theoretical computer science 217, 521
(2008).
[2] Sorva, J.: Notional machines and introductory programming education.
ACM Transactions on Computing Education (TOCE) 13(2), 8 (2013)
[3] Du Boulay, B., O'Shea, T., Monk, J.: The black box inside the glass box:
presenting computing concepts to novices. International Journal of Man-
Machine Studies 14(3), 237-249 (1981)
[4] Bonar, J., Soloway, E.: Preprogramming knowledge: A major source of
misconceptions in novice programmers. Human-Computer Interaction
1(2), 133-161 (1985)
[5] Eckerdal, A., Thun, M.: Novice java programmers' conceptions of object
and class, and variation theory. In: ACM SIGCSE Bulletin. vol. 37, pp.
89-93. ACM (2005)
[6] Lahtinen, E., Ala-Mutka, K., Jarvinen, H.M.: A study of the dificulties
of novice programmers. In: ACM SIGCSE Bulletin. vol. 37, pp. 14-18.
ACM (2005)
[7] Milne, I., Rowe, G.: Difculties in learning and teaching programming
views of students and tutors. Education and Information technologies
7(1), 55-66 (2002)
[8] Kaczmarczyk, L.C., Petrick, E.R., East, J.P., Herman, G.L.: Identifying
student misconceptions of programming. In: Proceedings of the 41st
ACM technical symposium on Computer science education. pp. 107-
111. ACM (2010)
[9] Ala-Mutka, K.M.: A survey of automated assessment approaches for
programming assignments. Computer science education 15(2), 83{102
(2005)
[10] Delev, T., Gjorgjevikj, D.: E-lab: Web based system for automatic
assessment of programming problems. Web proceedings ICT-
Innovations (2012).
[11] Venables, A., Haywood, L.: Programming students need instant
feedback! In: Proceedings of the 5fth Australasian conference on
Computing education-Volume 20. pp. 267-272. Australian Computer
Society, Inc. (2003)
[12] Rahman, K.A., Nordin, M.J.: A review on the static analysis approach in
the automated programming assessment systems. In: Proceedings of the
national conference on programming. vol. 7 (2007)
[13] Truong, N., Roe, P., Bancroft, P.: Static analysis of students’ java
programs. In: Proceedings of the Sixth Australasian Conference on
Computing Education-Volume 30. pp. 317325. Australian Computer
Society, Inc. (2004).
[14] Kremenek, Ted. "Finding software bugs with the clang static analyzer."
California: Apple Inc (2008).
[15] Lattner, C., Adve, V.: Llvm: A compilation framework for lifelong
program analysis & transformation. In: Code Generation and
Optimization, 2004. CGO 2004. International Symposium on. pp. 7586.
IEEE (2004)
[16] Fitzgerald, Sue, et al. "Debugging: finding, fixing and flailing, a multi-
institutional study of novice debuggers." Computer Science Education
18.2 (2008): 93-11
... These results seem to be course-specific, as they are not consistent with what has been observed in this work as will be discussed in Section 6. Liu and Peterson [28] also reported positive results when using PyTA in an introductory course, where they observed (compared to a previous year) a significant reduction in the number of repeated errors per submission, submissions to pass a programming exercise and submissions required to solve the most common errors. Other studies confirming the utility of using static analysis tools in introductory programming classes include Delev and Gjorgjevikj [35] (in C) and Schorsch [36] (in Pascal). ...
Article
Full-text available
Analyzing student coding data can help researchers understand how novice programmers learn and inform practitioners on how to best teach them. This work explores how using static analysis tools in programming assignments can provide insight into student behavior and performance. The use of three static analysis tools in the assignments of an introductory programming course has been analyzed. Our findings confirm previous work regarding that formatting and documentation issues are the most common issues found in student code, that this is constant regardless of major and performance in the course and that there are certain error types which are more correlated with performance. We also found that total error frequency in the course correlates with final course grade and that the presence of any kind of error in final submissions correlates with low performance on exams. Furthermore, we found females to produce less documentation and style errors than males and students who partner to produce less errors in general than students working alone. Our results also raise concerns on the use of certain metrics for assessing the difficulty of fixing errors by students.
... This tool was used in a CS1 course of 520 students and a qualitative analysis of students' opinions were positive overall and instructors teaching the course all noticed in an increased quality of student programs. Delev and Gjorgjevikj [5] report their findings on common errors of C source code written by novice programmers detected by two different static analysis tools. Clang Static Analyzer and CppCheck were both run 13,960 source code files and the desired data was parsed from their reports. ...
Conference Paper
Students learning to program often rely on feedback from the compiler and from instructor-provided test cases to help them identify errors in their code. This feedback focuses on functional correctness, and the output, which is often phrased in technical language, may be difficult to for novices to understand or effectively use. Static analyses may be effective as a complementary aid, as they can highlight common errors that may be potential sources of problems. In this paper, we introduce PyTA, a wrapper for pylint that provides custom checks for common novice errors as well as improved messages to help students fix the errors that are found. We report on our experience integrating PyTA into an existing online system used to deliver programming exercises to CS1 students and evaluate it by comparing exercise submissions collected from the integrated system to previously collected data. This analysis demonstrates that, for students who chose to read the PyTA output, we observed a decrease in time to solve errors, occurrences of repeated errors, and submissions to complete a programming problem. This suggests that PyTA, and static analyses in general, may help students identify functional issues in their code not highlighted by compiler feedback and that static analysis output may help students more quickly identify debug their code.
... Delev and Gjorgjevikj [3] analyze the results of running static analysis tools on student submissions and report that more than half of such submissions contain errors reported by these tools. They argue that the use of static analysis tools should be included in the learning process. ...
Conference Paper
Functional decomposition is the process of breaking a larger problem into smaller subproblems so that each component implements only a single task. Although functional decomposition is integral to computer science, it is of ten overlooked in introductory computer science education due to the challenges of teaching and assessing it given limited resources. Earthworm is a tool that leverages static analysis techniques to generate suggestions on how to improve the decomposition of Python programs. Each suggestion includes the lines of code that can be refactored into a new function, the arguments that must be passed to this function and the variables returned from the function. The tool is intended to be used in introductory computer science courses to help students develop decomposition skills.
Conference Paper
Full-text available
E-Lab is a system developed at Faculty of Computer Science and Engineering for solving and auto-grading programming problems from introduction to programming courses. The main goal is to simplify and improve the organization and the process of solving programming problems from large group of students in dedicated computer labs using centralized server. All the work from the students is done in a web browser using a web-based code editor and everything is stored, compiled and executed on the server. The system keeps records of all problem at-tempts from identified students which are used as attendance records. All the problems and solutions are under version control system (Git). The platform supports different types of problems in several programming languages (C, C++, Java) and it's designed to be easily extended.
Article
Full-text available
This article brings together, summarizes, and comments on several threads of research that have contributed to our understanding of the challenges that novice programmers face when learning about the runtime dynamics of programs and the role of the computer in program execution. More specifically, the review covers the literature on programming misconceptions, the cognitive theory of mental models, constructivist theory of knowledge and learning, phenomenographic research on experiencing programming, and the theory of threshold concepts. These bodies of work are examined in relation to the concept of a “notional machine”—an abstract computer for executing programs of a particular kind. As a whole, the literature points to notional machines as a major challenge in introductory programming education. It is argued that instructors should acknowledge the notional machine as an explicit learning objective and address it in teaching. Teaching within some programming paradigms, such as object-oriented programming, may benefit from using multiple notional machines at different levels of abstraction. Pointers to some promising pedagogical techniques are provided.
Article
Full-text available
We present a process model to explain bugs produced by novices early in a programming course. The model was motivated by interviews with novice programmers solving simple programming problems. Our key idea is that many programming bugs can be explained by novices inappropriately using their knowledge of step-by-step procedural specifications in natural language. We view programming bugs as patches generated in response to an impasse reached by the novice while developing a program. We call such patching strategies bug generators. Several of our bug generators describe how natural language preprogramming knowledge is used by novices to create patches. Other kinds of bug generators are also discussed. We describe a representation both for novice natural language preprogramming knowledge and novice fragmentary programming knowledge. Using these representations and the bug generators, we evaluate the model by analyzing four interviews with novice programmers.
Article
Full-text available
Tools based on static analysis can be used to find defects in programs. Tools that do shallow analyses based on pattern matching have existed since the 1980's and although they can analyze large programs they have the drawback of producing a massive amount of warnings that have to be manually analyzed to see if they are real defects or not. Recent technology advances has brought forward tools that do deeper analyses that discover more defects and produce a limited amount of false warnings. These tools can still handle large industrial applications with millions lines of code. This article surveys the underlying supporting technology of three state-of-the-art static analysis tools. The survey relies on information in research articles and manuals, and includes the types of defects checked for (such as memory management, arithmetics, security vulnerabilities), soundness, value and aliasing analyses, incrementality and IDE integration. This survey is complemented by practical experiences from evaluations at the Ericsson telecom company.
Conference Paper
Full-text available
Computing educators are often baffled by the misconceptions that their CS1 students hold. We need to understand these misconceptions more clearly in order to help students form correct conceptions. This paper describes one stage in the development of a concept inventory for Computing Fundamentals: investigation of student misconceptions in a series of core CS1 topics previously identified as both important and difficult. Formal interviews with students revealed four distinct themes, each containing many interesting misconceptions. Three of those misconceptions are detailed in this paper: two misconceptions about memory models, and data assignment when primitives are declared. Individual misconceptions are related, but vary widely, thus providing excellent material to use in the development of the CI. In addition, CS1 instructors are provided immediate usable material for helping their students understand some difficult introductory concepts.
Conference Paper
Full-text available
This paper describes the work that has been undertaken to improve the quantity and quality of feedback given to students studying introductory Java programming. An on- line submission program (submit) was developed that specifically addresses the needs of beginning programming students by providing automatic and instant feedback to a student's programming efforts. Traditionally, this feedback was provided by staff in scheduled laboratory sessions, yet the amount of individual attention a particular student received was haphazard. In addition, the introduction of the submit program has reduced the marking and administrative load associated with providing timely feedback to students for participating staff. submit is more than just an on-line letter box for students to post their programs. submit also provides instant feedback which encourages students to improve their programs to meet all given requirements as well as to improve programming style. In order to assess the effectiveness of the submit program over four trials, student responses to questionnaires were collated and support some tentative conclusions as to the effectiveness of submit as a learning aid to students. Staff opinion from those involved in the day-to-day teaching of Java programming was also solicited. These responses have been very positive, especially from the staff that supervise laboratory work . .
Article
Practical programming is one of the basic skills pursued in computer science education. On programming courses, the coursework consists of programming assignments that need to be assessed from different points of view. Since the submitted assignments are executable programs with a formal structure, some features can be assessed automatically. The basic requirement for automated assessment is the numerical measurability of assessment targets, but semiautomatic approaches can overcome this restriction. Recognizing automatically assessable features can help teachers to create educational models, where automatic tools let teachers concentrate their work on the learning issues that need student-teacher interaction the most.Several automatic tools for both static and dynamic assessment of computer programs have been reported in the literature. This article promotes these issues by surveying several automatic approaches for assessing programming assignments. Not all the existing tools will be covered, simply because of the vast number of them. The article concentrates on bringing forward different assessment techniques and approaches to give an interested reader starting points for finding further information in the area. Automatic assessment tools can be used to help teachers in grading tasks as well as to support students' working process with automatic feedback. Common advantages of automation are the speed, availability, consistency and objectivity of assessment. However, automatic tools emphasize the need for careful pedagogical design of the assignment and assessment settings. To effectively share the knowledge and good assessment solutions already developed, better interoperability and portability of the tools is needed.
Article
We have conducted a web-based questionnaire on the various concepts and topics of object-oriented programming that students on introductory courses found most difficult to cope with. A statistical analysis of our results shows that those topics that rely on a clear understanding of pointers and memory-related concepts (such as copy constructors and virtual functions) prove to be the most difficult. In other words, we believe these concepts are only hard because of the student's inability to comprehend what is happening to their program in memory, as they are incapable of creating a clear mental model of its execution. These results would suggest that a clearer approach to teaching these topics would be beneficial to students. We are currently working on a visualization-based approach to address these issues.
Article
Simplicity and visibility are two important characteristics of programming languages for novices. Novices start programming with very little idea of the properties of the notional machine implied by the language they are learning. To help them learn these properties, the notional machine should be simple. That is, it should consist of a small number of parts that interact in ways that can be easily understood, possibly by analogy to other mechanisms with which the novice is more familiar. A notional machine is the idealized model of the computer implied by the constructs of the programming language. Visibility is concerned with methods for viewing selected parts and processes of this notional machine in action. We introduce the term “commentary” which is the system's dynamic characterization of the notional machine, expressed in either text or pictures on the user's terminal. We examine the simplicity and visibility of three systems, each designed to provide programming experience to different populations of novices.