No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Effectiveness of online assessment
Abstract
For five academic years we have engaged in an on-going study of the effectiveness of online assessment of student programming abilities for introductory programming courses in Computer Science. Our results show that online evaluation can be implemented securely, efficiently, and can result in increased student motivation and programming efficacy; however, unless online components are integrated throughout the course evaluations, student competence will be underestimated. Our data reveals disadvantages of online evaluations, but also shows that both students and faculty benefit when online evaluations are implemented appropriately.
... This provision is meant to prevent students from failing the entire test if they get stuck on one small step. Provision of partial credit for incomplete programs and provision of hints during the test go a long way to alleviate the stress that researchers have associated with students taking online tests [3, 6, 20]. Finally, the tests are open-book, open-notes. ...
... In an evaluation of online tests as replacement of traditional (written) tests, researchers found that online tests can result in increased student motivation and programming efficacy [20], and that both students and instructors agree that online tests provide a more accurate measure of student efficacy. ...
... In earlier studies, researchers found that students tend to do poorly on the first online test [3, 6, 20] – so much so that researchers in these studies strongly recommended that multiple online tests be conducted in order to eliminate the stress associated with online testing. We wanted to find out whether the treatment, and in particular, closed labs, had any effect on the first test. ...
The majority of earlier studies have found no positive effect of closed labs on student performance or retention in Computer Science I. Since these studies used written tests to assess student performance, their results may have been affected by a mismatch between what was taught in closed labs and what was assessed in the written test. On the other hand, online tests that involve writing and debugging programs assess the very knowledge and skills taught in closed labs. So, we conducted a study to evaluate the effect of closed labs on student performance and retention when they are combined with online testing in the course. We found that even with online testing, closed labs did not significantly affect either the total score of the students in the course or retention in the course. On the other hand, students with closed or even optional labs scored significantly better on their online first test than students with open labs scored on their written first test. Since this is in spite of the fact that online tests are more stressful to students than written tests, closed labs helped students prepare for online tests. Closed labs, when combined with online testing, helped significantly improve the course performance of above-average students. Finally, there was significant and large correlation between Math SAT score and student performance in Computer Science I.
... By 2005 there was sufficient interest in the field of automatic assessment systems that multiple reviews were published [8], [9], highlighting the emergence of evidence that automatic assessment can lead to increased student performance [10], [11]. Another benefit realised with automatic assessment systems is greater ease in detecting plagiarism, tools for the purpose having been included in several of the systems surveyed. ...
... We briefly summarise and discuss these points: the most frequent student feedback is on the immediate feedback that the system provides. Some student comments mention explicitly the usefulness of the system's feedback which allows to identify the errors they have made more easily (items 10,11,12,18). In addition to these generic endorsements, some students mention explicitly advantages of the test-driven development such as re-assurance regarding correctness of code (item 14), quick feedback on refactoring (15), the indirect introduction of unit tests through the system (9), and help in writing clean code (16). ...
We describe a method of automatic feedback provision for students learning
programming and computational methods in Python. We have implemented, used and
refined this system since 2009 for growing student numbers, and summarise the
design and experience of using it. The core idea is to use a unit testing
framework: the teacher creates a set of unit tests, and the student code is
tested by running these tests. With our implementation, students typically
submit work for assessment, and receive feedback by email within a few minutes
after submission. The choice of tests and the reporting back to the student is
chosen to optimise the educational value for the students. The system very
significantly reduces the staff time required to establish whether a student's
solution is correct, and shifts the emphasis of computing laboratory student
contact time from assessing correctness to providing guidance. The self-paced
nature of the automatic feedback provision supports a student-centred learning
approach. Students can re-submit their work repeatedly and iteratively improve
their solution, and enjoy using the system. We include an evaluation of the
system and data from using it in a class of 425 students.
... Lab exams are used as a way of providing a valid assessment of programming efficacy. Students and staff perceive the assessment to be fair and an accurate assessment of programming skills [63,236,715] and to improve student motivation [52]. Students working on computers produce more accurate code than on paper [360]. ...
As computing becomes a mainstream discipline embedded in the school curriculum and acts as an enabler for an increasing range of academic disciplines in higher education, the literature on introductory programming is growing. Although there have been several reviews that focus on specific aspects of introductory programming,there has been no broad overview of the literature exploring recent trends across the breadth of introductory programming.
This paper is the report of an ITiCSE working group that conducted a systematic review in order to gain an overview of the introductory programming literature. Partitioning the literature into papers addressing the student, teaching, the curriculum, and assessment, we explore trends, highlight advances in knowledge over the past 15 years, and indicate possible directions for future research.
... Lab exams are used as a way of providing a valid assessment of programming efficacy. Students and staff perceive the assessment to be fair and an accurate assessment of programming skills [63,236,715] and to improve student motivation [52]. Students working on computers produce more accurate code than on paper [360]. ...
A broad review of research on the teaching and learning of programming was conducted by Robins et al. in 2003. Since this work there have been several reviews of research concerned with the teaching and learning of programming, in particular introductory programming. However, these reviews have focused on highly specific aspects, such as student misconceptions, teaching approaches, program comprehension, potentially seminal papers, research methods applied, automated feedback for exercises, competency-enhancing games, and program visualisation. While these aspects encompass a wide range of issues, they do not cover the full scope of research into novice programming. Some notable areas that have not been reviewed are assessment, academic integrity, and novice student attitudes to programming. There does not appear to have been a comprehensive review of research into introductory programming since that of Robins et al. It is therefore timely to conduct and present such a review in order to gain an understanding of the research focuses, to highlight advances in knowledge since 2003, and to indicate possible future directions for research. The working group will conduct a systematic literature review based on the guidelines proposed by Kitchenham et al. This research project is well suited to an ITiCSE working group as the synthesis and discussion of the literature will benefit from input from a variety of researchers drawn from different backgrounds and countries.
... The achievement of competences has to be evaluated appropriately by means of the definition of formal and reliable evaluation processes (Montero et al., 2014). In this regard, online evaluation can be implemented securely, efficiently, and can result in increased student motivation and programming efficacy (Woit and Mason, 2003), but distributed and distance learning are creating some of the biggest changes in the evaluation process (Carter et al., 2003). The development and assessment of professional skills in higher education programs is an issue that includes defining and assessing these skills in the contexts of either a subject or an entire program (Cajander et al., 2011). ...
Nowadays, competence based assessment is a term widely used in higher educational institutions. Many educational centres focus their efforts on the concretion of competences for the evaluation. For this purpose,
in the European Union (EU) was defined the European e-Competence Framework (e-CF), which provides a reference of 40 competences applied to the field of Information and Communication Technologies (ICT) using a common language of competences, skills and capability levels. On the other hand, the incorporation of Information Technologies (IT) such as Audience Response Systems (ARSs) in the European Higher Education Area (EHEA) or other educational institutions has supposed a new way to teach. Unifying these two concepts (Competences and ARSs), this paper presents a novel approach for automated measurement of competences applied to the field of Software Engineering and shows a description about the conceptual model to develop this type of evaluations. Furthermore, we propose the integration of SIDRA (ARS tool) and SECEIP (Software Engineering Competence Evaluation Internet Portal). This portal will be endowed with a new means of assessing students, thanks to the use of wireless devices, such as, smartphones or tablets.
... The achievement of competences has to be evaluated appropriately by means of the definition of formal and reliable evaluation processes (Montero et al., 2014). In this regard, online evaluation can be implemented securely, efficiently, and can result in increased student motivation and programming efficacy (Woit and Mason, 2003), but distributed and distance learning are creating some of the biggest changes in the evaluation process (Carter et al., 2003). The development and assessment of professional skills in higher education programs is an issue that includes defining and assessing these skills in the contexts of either a subject or an entire program (Cajander et al., 2011). ...
Abstract.- This research provides analysis of existing models for competence evaluation and proposes a software engineering competence remote evaluation process model. We separate competences into two domains: professional and academic. The model is developed and implemented for e-CF professional competences and academic competences evaluation for Software Engineering Master Program graduates. The remote evaluation process is based on a proposed methodology in four steps. The remote evaluation process consists of two business processes: (a) Competence test design, and then (b) Competence test completion. Business Process Model and Notation (BPMN) model was developed as a summary of both processes.
Keywords.- Software Engineering Competence Evaluation Internet Portal, The European e-Compe-tence Framework (e-CF), competence-based education.
... In addition to having to master software testing skill at a higher level, having to use an automatic assessment tool itself may pose additional challenges. Woit and Mason [2003] noticed that their students needed experience to work successfully with their system in an online examination situation. ...
This is a position paper in which I argue that massive open online programming courses can benefit by the application of automated assessment of programming assignments. I gathered success factors and identified concerns related to automatic assessment through the analysis of experiences other researchers have reported when designing and using automated assessment of programming assignments and interpret their potential applicability in the context of massive open online courses (MOOCs). In this paper I explain the design of our own assessment software and discuss our experience of using it in relation to the above-mentioned factors and concerns. My reflection on this experience can inform MOOC designers when having to make decisions regarding the use of automatic assessment of programming assignments.
... Järvinen [3] note the tendency of students to try to minimize their workload, which renders optional activities in the learning process ineffective. Woit and Mason [144] carried out a five year study comparing different assignment strategies, where two experiments contained optional assignments. Along the same lines, they reported that their students mostly ignored optional tasks, which was also directly reflected in poor midterm exam results. ...
... Grading in online settings has also been studied in the attempt to reduce time spent on grading by faculty [24]. Furthermore, several researchers have shared their online teaching practice experience [27], making a strong point with respect to the benefit of this type of training. ...
This paper describes an online application for test design and evaluation of trainees. We address two main topics: activity flow modeling and adaptive test design. With activity flow modeling, we achieve high usability and structural coherence, while the adaptive test design method that we propose facilitates dynamic generation of tests based on topic relevance. Our method ensures the creation of adaptive tests, targeted to specific topics of interest for users, and employs specific policies to adjust the difficulty level of tests. We also describe a taxonomy-based test design method, that enhances the testing environment with new usability and efficiency attributes.
... ost. Therefore many of the questions gave three different options and asked the students to input the cost or benefit of each and then input which was the most appropriate. This meant that it was much easier for students to get part marks as most parts of the question were independent. The final decision was usually only allocated 10% of the marks. Woit and Mason (2003) found that typically the first online assessment in a course would have lower marks then subsequent assessments. This was found in this course. The level of difficulty of each assignment was fairly similar but the average student results in chronological order were 67%, 87%, 83% and 86% showing a dramatic jump between the first two assi ...
Quantitative Engineering courses typically require students to solve complex mathematical problems with many parts all based on the same data. Students should be able to identify as well as apply the correct mathematics for each step. When assessing student answers it is common to give part marks for correct calculations, even if the answer is incorrect. If the first part of the answer is wrong, but subsequent parts are correct, marks should be awarded for the correct parts. This paper describes the use of two online quiz systems in several engineering courses and discusses how effectively they accommodate the requirements above. End of course survey results show that students prefer online assessment to traditional hard copy assignments. Comparison of online assessment results with similar questions assessed traditionally show higher results, indicating that online assessments encourage students to put more effort into obtaining the correct answer.
... Many articles on this topic of computer aided learning and assessment can be found in the Electronic Journal for e- Learning, for example [2]. Work studying the effectiveness of computer-based assessment for introductory programming courses in C, Unix and Scheme can be found in references [3] and [4]. Costagliota and Fucella [5] review on-line testing and a discuss state of the art Learning Management Systems. ...
We have developed an e-learning and e-assessment tool for a first year university course on computer programming. The tool presents information and questions to the student and provides immediate feedback to the student regarding correctness and score. An answer may be textual, or a selection from multiple choice, or a complete computer program involving several files. When a program is required, the answer is open-ended in the sense that the contents of the program are only checked in terms of functionality. That is, the program must meet the functional specifications given in the question. The results are packaged for the instructor in a specially-formatted file that contains the questions as they were presented to the student, the student's answers, and an automated assessment. The tool is platform-independent, since it is written in Java with no particular operating system dependencies. It is capable of shuffling questions, presenting a random subset of a group of questions, and awarding partial credit for repeated tries or case mismatches (where case is important). The tool includes security features to improve evaluation integrity, including encryption of answers within the tool and encryption of student results.
... It is not uncommon that several students, upon completing a year of study on programming, exhibit serious shortcomings on basic programming knowledge (McCracken et al. 2001). It was found that students with little or no practical work were able to produce a piece of code in the final traditional way of assessment through memorization and achieve a 'good' grade in the course (Woit & Mason 2003). Furthermore, it is difficult to closely observe the progress of a particular student, especially in large classes. ...
Abstract This paper presents Programming Adaptive Testing (PAT), a Web-based adaptive testing system for assessing students' programming knowledge. PAT was used in two high school programming classes by 73 students. The question bank of PAT is composed of 443 questions. A question is classified in one out of three difficulty levels. In PAT, the levels of difficulties are adapted to Bloom's taxonomy lower levels, and students are examined in their cognitive domain. This means that PAT has been designed according to pedagogical theories in order to be appropriate for the needs of the course ‘Application Development in a Programming Environment’. If a student answers a question correctly, a harder question is presented, otherwise an easier one. Easy questions examine the student's knowledge, while difficult questions examine the student's skills to apply prior knowledge to new problems. A student answers a personalized test composed of 30 questions. PAT classifies a student in one out of three programming skills' levels. It can predict the corresponding classification of students in Greek National Exams. Furthermore, it can be helpful to both students and teachers. A student could discover his or her programming shortcomings. Similarly, a teacher could objectively assess his or her students as well as discover the subjects that need to be repeated.
... The exercises were not prescribed, only voluntary students try them. As already reported (Woit and Mason, 2003), students did not volunteer easily. Apart the initial curiosity of the first week, we only noted the usual peak during the week that precede examina- tions. ...
Mechanised grading is now mature. In this paper, we propose elements for the next step: building a commodity infrastructure for grading. We propose --- an architecture for a grading component able to be embedded into various learning environments, --- a set of extensible Internet-based protocols to interact with this component, and --- a self-contained file format to thoroughly define an exercise in order to ease deployment. The infrastructure was designed to be scalable, robust and secure. First experiments with a partial implementation of this platform show the versatility and the neutrality of the grading component. It does not impose any IDE and it respects the tenets of the embedding learning environment or course-based system.
... The exercises were not prescribed, only voluntary students try them. As already reported in [20] , students did not volunteer easily. Apart the initial curiosity of the first week, we noted the usual peak during the week that precede ex- aminations. ...
Mechanised grading is now mature: lots of experiments proved it. In this paper, we propose elements for the next step: standardisation. We propose --- an architecture for a grading component able to be embedded into various learning environments, --- a set of extensible Internet-based protocols to interact with this component, and --- a self-contained file format to thoroughly define an exercise in order to ease deployment. The architecture was designed to be scalable, robust and secure. First experiments with a partial implementation of this platform show the versatility and the neutrality as well of the grading component. It does not impose an IDE, it respects the tenets of the embedding learning environment or course-based system.
This paper investigates the impact of layers of quizzing on student performance on subsequent tests and the final exam. The first layer of quizzes are on-line "pre-quizzes" designed to prepare students for follow-on classroom discussions. The second layer quizzes are short "post-quizzes" that are given during class time after discussing questions about the material covered by pre-quizzes. A study involving fifteen semesters of data and 348 students indicated that students performed better on tests and the final exam with this quizzing strategy compared to a more traditional homework based approach. These results agree with the body of research that indicates that testing enhances learning. Significantly, we found an 8 point pass rate increase on the Computer Systems I final exam and a 10 point pass rate increase on the Systems II final.
This research reports the development of an online three-tier diagnostic instrument for pre-university students related to cellular respiration. To date, only few studies have been conducted to identify students? alternative conception in the topic of cellular respiration. One of the contributing factors is due to lack of instrument developed to measure these alternative conceptions. Three-tier tests are used as assessment tools for lecturers to determine students? alternative conceptions related to cellular respiration and their knowledge and understanding about the concepts. The research incorporates both qualitative and quantitative methods. The instrument was developed in five phases: (1) construction of items, (2) pilot study, (3) validation of instrument, (4) transforming the instrument into an online assessment tool, and (5) the administration of the Online Diagnostic Tool (ODiT). The Cellular Respiration ODiT consists of three tiers: answer and reasoning tiers to measure ?content knowledge? and ?explanatory knowledge? respectively, and a third tier that measures the level of confidence of pre-university students. Analysis of the students? responses demonstrated acceptable reliability and validity of the instrument. The research also shows that both students and lecturers benefit when online diagnostic tests are implemented appropriately.
We present an e-learning and e-assessment aid for studying and teaching programming languages (Java, C, Perl, Ruby and Python). The student uses the wizard to go through a series of pages which present information, link to web resources, ask questions, and mark answers. The questions can be of short-answer type, multiple choice or programming. In the short-answer questions, hints can be automatically generated in the form of scrambled spelling or first letters and lengths of the words in the answer. In programming questions, hints may be in the form of skeletons of programs with blank “to do” regions which are to be filled in. Programs submitted by the student are compiled, run and checked against expected output. The wizard can present a randomly selected and shuffled subset of the questions taken from a question repository. The repository contains flags determining the hints and format of each question. A record of the session, containing the questions and possibly multiple attempts at solutions from the student, is saved in an encrypted format. The answer key within the wizard is also encrypted so that answers cannot be obtained by reverse engineering.
This paper describes a simple Unix-based repository that supports programming intensive courses. Initially motivated by the need to capture sample artifacts for the capstone course sequence, the repository has evolved to support progam submission, on-line testing, and automated program grading. Repository features enforce naming conventions, assignment due dates, while supporting the use of Unix in programming classes. The repository also facilitates consistent on-line program compilation and execution. Repository utilities are provided to support first-time adoption, for archiving repositories from past terms, and for establishing repositories for a new academic term. Results from the transfer of the technology to first-time adopters are presented.
This paper presents a comparison study between a traditional pen & paper examination and an online exam according to student's opinions. Although a very current topic, it is the first such comparison documented in the scientific literature. In this study an online exam refers to a location independent online examination in a university setting. E-learning is an approach to learning, in which the new information and communication technology is utilized to enhance the learning processes. The advances in technology and applications offer enormous possibilities to education in terms of effectiveness and flexibility for both students and teachers. These new possibilities are already changing the structures and methods of teaching in universities. Exams based on rote memory and conducted in a large classroom with only pen and paper as tools and all communication strictly forbidden have been predominant in Finnish universities since the Second World War. Although the theory of learning has developed significantly and demonstrated such exams ineffective for learning, this way of evaluating students has kept its position in higher education. Online exams are becoming popular in online courses, but have not been widely implemented in universities. The exam this study was built around was organized for the course "Introduction to Digital Media" of the Department of Information Processing Sciences in University of Oulu. There is no novelty in the web exam itself, but in the comparison of the examination methods according to the students, teacher and the theory of learning. In addition to the 235 feedbacks derived from the exam I have interviewed the teacher of the course in question. Online exams prove to be a step towards a student-centered and constructivist learning.
This article reports on the result of a study of optimizing programming assignments based on data collected through the use of a self-contained assignment generation and submission system. Using data collected by the system, we were able to increase the student success rate by modifying the course structure after identifying student bottlenecks. The value of the study is in demonstrating how collecting online statistics of student usage may enable the iterative optimization of course delivery.
To help students gauge their understanding of basic concepts and encourage good study habits, we administered short online quizzes that use generic questions in the crucial first few weeks of a course. The purpose of the study was to investigate whether the combination of these web quizzes with generic questions with only binary feedback (right or wrong) would be beneficial for students' learning. We implemented these quizzes in three classes in two different subjects at two different universities, one in Sweden and one in the USA. The students' views on the quizzes' effect on their learning was investigated with surveys and interviews.Almost all students appreciated having these quizzes and 38% of them changed their view on how much they knew of the material covered in the course. Furthermore, over 20% of the students reported altering their study habits as a consequence, in particular studying harder or earlier.In conclusion, this study indicated that short quizzes using generic questions with limited correct/incorrect feedback on each question, have positive effects when administered early in courses.The combination of generic questions and short quizzes could be of value for those contemplating automatic formative assessment, particularly if there is some hesitation with respect to the resources needed for constructing and validating the automatic feedback.
TRAKLA2 is an online practicing environment for data structures and algorithms. The system includes visual algorithm simulation exercises, which promote the understanding of the logic and behaviour of several basic data structures and algorithms. One of the key features of the system is that the exercises are graded automatically and feedback for the
learner is provided immediately.
The system has been used by many institutes worldwide. In addition, several studies conducted with the system have revealed that the learning results are similar to those obtained in closed labs if the tasks are the same. Thus, automatic assessment of visual algorithm simulation exercises provides a meaningful way to reduce the workload of grading the exercises while still maintaining good learning results.
Practical programming is one of the basic skills pursued in computer science education. On programming courses, the coursework consists of programming assignments that need to be assessed from different points of view. Since the submitted assignments are executable programs with a formal structure, some features can be assessed automatically. The basic requirement for automated assessment is the numerical measurability of assessment targets, but semiautomatic approaches can overcome this restriction. Recognizing automatically assessable features can help teachers to create educational models, where automatic tools let teachers concentrate their work on the learning issues that need student-teacher interaction the most.Several automatic tools for both static and dynamic assessment of computer programs have been reported in the literature. This article promotes these issues by surveying several automatic approaches for assessing programming assignments. Not all the existing tools will be covered, simply because of the vast number of them. The article concentrates on bringing forward different assessment techniques and approaches to give an interested reader starting points for finding further information in the area. Automatic assessment tools can be used to help teachers in grading tasks as well as to support students' working process with automatic feedback. Common advantages of automation are the speed, availability, consistency and objectivity of assessment. However, automatic tools emphasize the need for careful pedagogical design of the assignment and assessment settings. To effectively share the knowledge and good assessment solutions already developed, better interoperability and portability of the tools is needed.
A recent study [7] has shown that many computing students are not able to develop straightforward programs after the introductory programming sequence. Normal student assessment should have highlighted this problem, it did not, therefore normal assessment of programming ability does not work.We examine why current assessment methods (written exams and programming assignments) are faulty. We investigate another method of assessment (the lab exam) and show that this form of assessment is more accurate.We explain why accurate assessment is essential in order to encourage students to develop programming ability.
Course evaluations are an important instrument for assessing teaching effectiveness and maintaining the quality of our academic programs. At Indiana University South Bend students are given the opportunity to evaluate their courses at the end of each semester. The results are collected and analyzed by the corresponding academic units. The university dedicates a significant amount of monetary and human resources to conduct effective and confidential end-of-semester evaluations. In order to improve the speed, accuracy, anonymity, and privacy of course evaluations the authors have developed a web-based course evaluation system called IU-EVAL. To date, the system has been tested by approximately 23,500 students. In this paper we explore the motivations behind the development of IU-EVAL and discuss the lessons learned during the deployment and administration of the system. We will also share the results of our user survey.
The emergence of networked computers has brought to the forefront the use of web-based education, which defies boundaries. The pivotal advantage associated with web based learning is the classroom and platform independence it provides. Current web-based systems are found to be a collection of static pages accessorized with few videos. The learning associated with these web based systems is static and serial, making the whole process uninteresting and incomplete. The need for the hour is a Learning Management System (LMS), which adapts itself according to the learner's performance and learning path. This paper provides insight into building an effective and efficient LMS using five general principles, which are defined as the Five Elements. This leads to the creation of a dynamic, learnercentered environment which focuses on learning retention and cognitive development, juxtaposed with emerging technologies.
Practical exercises and assignments are an integral part of programming course, which is a core basic skill required in computer
science and best learned by doing. For effective learning, assignments have to be inspected and graded carefully but quickly
so that students may benefit from the useful and timely feedback. This can be facilitated by automating the process using
a specifically designed software system. PASS is such a system being developed in our department. A first version has been
pilot run, with very encouraging responses from tutors and students. We observed that with this automated system, students
can have their programs tested anytime, anywhere. They are encouraged to practice more without hesitation or embarrassment
about their programming mistakes, and thus this learning environment can boost up their motivation to continue the practice
of their programming skills. This paper aims at sharing the initial experiences we gained in using this system.
This paper describes a simple Unix-based repository that supports programming intensive courses. Initially motivated by the need to capture sample artifacts for the capstone course sequence, the repository has evolved to support progam submission, on-line testing, and automated program grading. Repository features enforce naming conventions, assignment due dates, while supporting the use of Unix in programming classes. The repository also facilitates consistent on-line program compilation and execution. Repository utilities are provided to support first-time adoption, for archiving repositories from past terms, and for establishing repositories for a new academic term. Results from the transfer of the technology to first-time adopters are presented.
This dissertation argues that automated assessment systems can be useful for both students and educators provided that the results correspond well with human markers. Thus, evaluating such a system is crucial. I present an evaluation framework and show how and why it can be useful for both producers and consumers of automated assessment systems. The framework is a refinement of a research taxonomy that came out of the effort to analyse the literature review of systems based on Latent Semantic Analysis (LSA), a statistical natural language processing technique that has been used for automated assessment of essays. The evaluation framework can help developers publish their results in a format that is comprehensive, relatively compact, and useful to other researchers. The thesis claims that, in order to see a complete picture of an automated assessment system, certain pieces must be emphasised. It presents the framework as a jigsaw puzzle whose pieces join together to form the whole picture. The dissertation uses the framework to compare the accuracy of human markers and EMMA, the LSA-based assessment system I wrote as part of this dissertation. EMMA marks short, free text answers in the domain of computer science. I conducted a study of five human markers and then used the results as a benchmark against which to evaluate EMMA. An integral part of the evaluation was the success metric. The standard inter-rater reliability statistic was not useful; I located a new statistic and applied it to the domain of computer assisted assessment for the first time, as far as I know. Although EMMA exceeds human markers on a few questions, overall it does not achieve the same level of agreement with humans as humans do with each other. The last chapter maps out a plan for further research to improve EMMA.
Assessment provides means for influencing students' learning results and strategies on university courses. However, supporting students to learn practical programming skills requires carefully designed assignments and their assessment. This paper studies challenges and approaches for assessing homework programming assignments. We present and discuss a semi-automatic assessment system for supporting programming assignments on large courses.
We have experimented with the use of weekly on-line quizzes to enhance student learning in our first-year computer science courses. In our experiments we compared the effectiveness of using quizzes to the alternative of using weekly marked laboratory assignments. The results of our experiments show that student learning and retention increase with on-line quizzes. Weekly quizzes would be impossible if they were administered and marked in the traditional fashion; thus, we developed and used a secure, online environment for administering, writing, and marking the quizzes, with most of the marking performed automatically via simple marking programs. In this paper we describe our experiment, present our observations about student learning, outline student opinion, relate problems we encountered and our solutions, and provide technical details of our closed-quiz and marking environment.
We have been using a combination of conventional and on-line programming examinations for the past 3 years in our first year programming courses. We plan to continue using on-line components because of the positive feedback from both students and faculty. We describe the operational and pedagogical lessons we have learned, illustrating problems encountered and our solutions, in the hope that others considering on-line examinations may also benefit from our experiences.
On-line programming tests and examinations were administered to approximately 120 first year computer science students in order to evaluate their practical skills. We describe our motivation for on-line testing, outline the technical details of our closed testing environment, and present our observations about student performance. We also compare the effectiveness of on-line tests versus conventional tests, report the problems we encountered and our solutions, relate student opinion regarding the on-line testing experiment, and present some insights gained by this experiment.
A study of cheating and plagiarism using a scenario based format, has provided an insight into attitudes towards questionable work practices of first year information technology students' within the School of Computer Science and Software Engineering of Monash University, and the School of Information Technology of Swinburne University. Students at both institutions showed similar responses to a range of cheating behaviours, in line with other literature. Plagiarism and cheating are widely tolerated and commonly practised, at least on the lower end of the spectrum of seriousness. However there were some areas of significant difference between the two student samples that warrant further research to develop strategic approaches for limiting cheating practices.
We have experimented with the use of weekly on-line quizzes to enhance student learning in our first-year computer science courses. In our experiments we compared the effectiveness of using quizzes to the alternative of using weekly marked laboratory assignments. The results of our experiments show that student learning and retention increase with on-line quizzes. Weekly quizzes would be impossible if they were administered and marked in the traditional fashion; thus, we developed and used a secure, online environment for administering, writing, and marking the quizzes, with most of the marking performed automatically via simple marking programs. In this paper we describe our experiment, present our observations about student learning, outline student opinion, relate problems we encountered and our solutions, and provide technical details of our closed-quiz and marking environment.
Examined perceptions of cheating held by 480 college students and 48 faculty members at 2 small colleges. Eighty-nine percent of students surveyed admitted they had cheated. Attitudinal variables were better at predicting cheating than were background variables. Faculty agreed on a definition of cheating yet did not agree on how to deal with cheating. (Author/NB)
The incidence and causes of cheating were investigated using a questionnaire, consisting of 21 cheating behaviors, which was distributed to students at an English university. Respondents were asked to indicate, confidentially, which of the behaviors they had engaged in. Reported cheating was widespread and some types of cheating (e.g., on coursework) were more common than others. Reported cheating was more common in men than women; more common with less able students than more able ones; more common in younger students than mature ones; and more common in science and technology students than those in other disciplines. It is suggested that students' motivation, in particular whether they are studying to learn rather than simply to obtain good grades, is a major factor in explaining these differences. The results also indicate that cheating consists of a number of different types of behavior rather than being a unitary concept. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
In this chapter, the authors present research findings concerning student academic dishonesty and discuss useful processes and sanctions in adjudicating cases. Prevention strategies and ways to implement proactive campus initiatives are also presented.
Through the use of a 49-item questionnaire administered to 380 university students, we investigated student cheating on exams, quizzes, and homework assignments. More than half the students reported cheating during the academic year on at least one of the above. The purpose of this paper was to uncover fundamental factors underlying cheating behavior. Through the use of correlational and factor analysis, three primary factors were identified: student immaturity, lack of commitment to academics, and neutralization. We offer interpretations of these factors and suggestions for testing these and other factors in future research.
One of the largest challenges facing educators teaching courses with a significant programming component is deciding how to evaluate each student's programming ability. In this paper we discuss how we have addressed this challenge in an introductory computer science course and statistically analyze the results to examine potential inequities in our approach.
Degree of dishonour . The Age
- D J Schemo
- Schemo D.J.
Web-based evaluation for the convenience of students, markers, and faculty
- D Mason
- D Woit
- A Abdullah
- H Barakat
- C Pires
- D Souza