Question
Asked 8th May, 2013

What tools do you use for automated grading of assignments that involve programming?

I teach several computer science courses that involve programming in C, Assembly, even Verilog and other languages. The courses are about programming, operating systrems, digital design, wireless sensor networks, and so on. Some I grade by hand, some using Makefiles, shell scripts or other tools. I was wondering what are the tools that you use for automated grading, that may include running programms in a sandbox and checking the output with usin regular expressions, or even for automated advising on coding practices, for, say, a class of 10 to 200 or more.

Most recent answer

We also use Web-CAT for (Java and Python), we have been using it for a couple of years now. It is open source works both on windows and linux and provides Eclipse plugins for electronic submission and feedback services directly from the Eclipse IDE. It can be integrated with Moodle as well.  This application has helped us to provide consistent and timely feedback to students. Its grading is hash, which may upset students at first, but it makes them better developers.

Popular answers (1)

10th May, 2013
Stephen H. Edwards
Virginia Tech (Virginia Polytechnic Institute and State University)
For Java, Python, C++, and a few other languages, we use Web-CAT (http://web-cat.org/), which is one of the most widely used open-source tools specifically for program grading. For C, assembly, and other low-level programming, it doesn't offer plug-ins so you can't use it there without doing some work.
6 Recommendations

All Answers (24)

10th May, 2013
Alexandre Miguel Pinto
Signal AI
We use Mooshak in several courses and it works pretty well.
2 Recommendations
10th May, 2013
Stephen H. Edwards
Virginia Tech (Virginia Polytechnic Institute and State University)
For Java, Python, C++, and a few other languages, we use Web-CAT (http://web-cat.org/), which is one of the most widely used open-source tools specifically for program grading. For C, assembly, and other low-level programming, it doesn't offer plug-ins so you can't use it there without doing some work.
6 Recommendations
12th May, 2013
Corrado Mencar
Università degli Studi di Bari Aldo Moro
Basically, I use Eclipse in my Java Programming lab. I use JUnit to evaluate student codes (in essence, I write a testcase and I verify if the testcase ends with success), the Checkstyle plugin to verify the code style (compliance to Sun Code Conventions). Finally, I use the Eclipse views to verify the compliance of code to some specifications required in the assignments.
1 Recommendation
12th May, 2013
Jacqueline L. Whalley
Auckland University of Technology
There has been some research done that investigates the approach that Corrado mentions in Australia by Cardell-Oliver see http://www.csse.uwa.edu.au/~rachel/ and Java programming. You might find the papers interesting given your question.
2 Recommendations
13th May, 2013
Stephen H. Edwards
Virginia Tech (Virginia Polytechnic Institute and State University)
@Corrado Mencar, I agree that the tools you mention can be very helpful. That's why Web-CAT (a) allows submissions directly from within Eclipse using a simple plug-in (also from a few other IDEs, or directly via a web browser), (b) allows instructors to use JUnit tests to check student solutions, (c) allows instructors to use Checkstyle and PMD to perform static analysis checks on student solutions, (d) allows students to write and submit their own JUnit tests (instructors can even require it), and (e) uses Clover to measure code coverage from student-written tests and give students feedback about which parts of their code are untested (good when instructors require students to test their own code). This is all for Java, of course--the open-source analysis ecosystem isn't equally mature for all programming languages.
2 Recommendations
13th May, 2013
Corrado Mencar
Università degli Studi di Bari Aldo Moro
@Stephen seems to be a very interesting tool. Thanks.
22nd May, 2013
Chris Duke
National University of Ireland, Galway
I've used Web-CAT to examine Java programming assignments and it is excellent, especially the integration with Eclipse and BlueJ. From recollection I think it requires a relatively old version of Tomcat to get it up and running. If you want to get up and running quickly I have a private AMI for Amazon EC2 that I would be happy to make public if there was any interest in it.
22nd May, 2013
Stephen H. Edwards
Virginia Tech (Virginia Polytechnic Institute and State University)
Thanks for the endorsement, Chris. On our main server, Web-CAT is running on the latest yum-available version of Tomcat (on CentOS), which I believe is 6.x.
22nd May, 2013
Chris Duke
National University of Ireland, Galway
Very happy to Stephen! The whole Tomcat thing could well have been me hitting a limit in my Linux skills when I was trying it out a couple of years ago on Ubuntu server :)
5th Jun, 2013
Leo Selavo
Elektronikas un datorzinātņu institūts
Just learned about an automatic grading tool for programming assignments coming from MIT and Microsoft Research:
The system grew out of work on program synthesis. The authors (Rishabh Singh, Solar-Lezama and Sumit Gulwani) will present a paper about this at the Association for Computing Machinery’s Programming Language Design and Implementation conference.
This might be useful for MOOCs. TAs at MIT already are using the system.
14th Jan, 2015
Scott Overmeyer
Southern New Hampshire University
Has Web-CAT been integrated with Moodle?
17th Jan, 2015
Stephen H. Edwards
Virginia Tech (Virginia Polytechnic Institute and State University)
We tried Moodle integration in the past, but Moodle was too much of a moving target.  Web-CAT supports grade exporting in a Moodle-compatible CSV format that is easy for uploading into Moodle's gradebook.  Also, we're considering supporting LTI v.2 for better integration with other LMSes, but I believe Moodle only supports LTI v1 (addresses single sign-on, but not gradebook integration).
1 Recommendation
17th Jan, 2015
Scott Overmeyer
Southern New Hampshire University
Thanks for your reply.  We need to integrate something for partial evaluation of code snippets as well as grading full submissions.
Are Bob & Bev Williges and Debbie Hix and Rex Hartson still at Tech?
21st Jan, 2015
Stephen H. Edwards
Virginia Tech (Virginia Polytechnic Institute and State University)
I'm not sure what you mean by "partial evaluation".  While there are some systems that handle evaluating code for small-scale coding questions (usually where you write a single function/method), and there are also systems that handle full-scale programming assignments, I'm not aware of a system that does both and that can be easily integrated into an LMS.  I'm just not sure if that's what you're targeting, though.  I'm not aware of anything that is widely used that has deeper moodle integration support in terms of moodle's activities, but I'd like to know of anything you find.
21st Jan, 2015
Scott Overmeyer
Southern New Hampshire University
I'll let you know if I find (or build) an all-purpose evaluator that can evaluate both partial and full programs (and be configured easily).
29th Sep, 2015
Nikolay Vyahhi
Bioinformatics Institute
We use Stepic.org to create and host programming assignments. More info how to create them is here – https://stepic.org/lesson/Step-Code-9173
4 Recommendations
31st Aug, 2016
Dirk Grunwald
University of Colorado
There are two grading systems I've come across in the last year worth looking at - one is http://www.inginious.org which uses Docker to have arbitrary code evaluation, multiple languages, etc. It's easy to install, has LDAP authentication and Moodle LTI integration including grade reporting. Short video showing an extension (using c-tutor) is at https://www.youtube.com/watch?v=l2hn_xq6B_g -- they don't have a video tutorial, but good docs.
Another is 'autolab' https://github.com/autolab - I haven't installed this yet, but the group of folks maintaining it at CMU are good. They also have an LTI interface, but haven't used it.
We explored using Web-Cat but the student involved didn't have the background in WebObjects. It would be useful to package this up as a Docker image so people can deploy it with less fuss.
8th Sep, 2016
Nishtha Dalal
Drexel University
After having researched grading practices for programming courses, my senior capstone focused on streamlining this process. This ultimately yielded in a full-scale web based platform -  AutoGradr.com which runs student code against test cases set up by professors. It's language agnostic, supports programs that read from/write to file, supports drag and drop of code (multiple files), integrates with Git 
Drexel's CS department has been using it  as the primary grading and submission tool for introductory C++ courses, and data structures courses. 
Professors can request free access anytime through the link, and get started immediately 
23rd Dec, 2016
Burcu Yildiz
Turkish-German University
I just started using stepik.org too.
It is easy to use and like it that we can integrate assignments into lessons. That way, I can lead the students to the assignment, providing them with the content they need to solve it right on the same platform. 
My only problem is that the grading is absolute. If I give 10 input/output pairs, I would  like the students to see for which of these pairs their code passed or failed and be graded accordingly. But right now, it gives them either 100% or 0 points.
 @Nikolay: do you have the same issue as well?
1 Recommendation
23rd Dec, 2016
Burcu Yildiz
Turkish-German University
Another tool we are trying to set up and use is AutoLab. I think it will have more options to tailor it to our needs than stepik, but the setting up is definitely harder.
We also use Web-CAT for (Java and Python), we have been using it for a couple of years now. It is open source works both on windows and linux and provides Eclipse plugins for electronic submission and feedback services directly from the Eclipse IDE. It can be integrated with Moodle as well.  This application has helped us to provide consistent and timely feedback to students. Its grading is hash, which may upset students at first, but it makes them better developers.

Similar questions and discussions

Related Publications

Article
The 1991 SIGCSE Technical Symposium was held March 7-8, 1991, in San Antonio, Texas. A post-symposium evaluation form was made available to all attendees, with the request that it be filled out to assist the '91 committee in evaluating the current symposium and to help future committees in planning their symposia. This article highlights the respon...
Got a technical question?
Get high-quality answers from experts.