Conference PaperPDF Available

Assessment and Support for Software Capstone Projects at the Undergraduate Level: A Survey and Rubrics

Authors:

Abstract and Figures

Software engineering and computer science students conduct a capstone project during the final year of their degree programs. These projects are essential in validating that students have gained required knowledge and they can synthesize and use that knowledge to solve real world problems. However, the external requirements on educational programs often do not provide detailed guidelines for how to conduct or support these capstone projects, which may lead to variations among universities. This paper presents the results from a survey conducted at 19 different Pakistani universities of the current management practices and assessment criteria used for the capstone project courses at Undergraduate level. Based upon the results of this survey and similar work on Master Thesis capstone projects in Sweden, we present assessment rubrics for software-related undergraduate capstone projects. We also present recommendations for the continuous improvement of capstone projects.
Content may be subject to copyright.
Assessment and Support for Software Capstone
Projects at the Undergraduate Level:
A Survey and Rubrics
Ehsan Ahmad
School of Computer Science and Technology
Northwestern Polytechnical University
710072 Xi’an, China
Email: ehsan@mail.nwpu.edu.cn
Bilal Raza
Blekinge Institute of Technology
SE-372 25 Ronneby, Sweden
Email: bilalraza001@gmail.com
Robert Feldt
Division of Software Engineering
Chalmers University of Technology
SE-412 96 Gothenburg, Sweden
Email: robert.feldt@chalmers.se
Abstract—Software engineering and computer science students
conduct a capstone project during the final year of their degree
programs. These projects are essential in validating that students
have gained required knowledge and they can synthesize and use
that knowledge to solve real world problems. However, the exter-
nal requirements on educational programs often do not provide
detailed guidelines for how to conduct or support these capstone
projects, which may lead to variations among universities. This
paper presents the results from a survey conducted at 19 different
Pakistani universities of the current management practices and
assessment criteria used for the capstone project courses at
Undergraduate level. Based upon the results of this survey and
similar work on Master Thesis capstone projects in Sweden, we
present assessment rubrics for software-related undergraduate
capstone projects. We also present recommendations for the
continuous improvement of capstone projects.
Index Terms—Capstone Project, Thesis, Survey, Management
Practices, Assessment Rubrics, Undergraduate Level
I. INTRODUCTION
A capstone or thesis project (Project) is an important part
of most education programs especially within engineering at
the undergraduate level. The main purpose of these projects is
to encourage students to apply the knowledge acquired during
their studies, individually or in a group formed as a team [1],
[4], [5]. Students are also expected to show how good they are
in solving real world problems of limited scope, with certain
constraints for the outcome-based evaluation suggested by
ACM/IEEE Computing Curricula 2004 [6]. Through research
and discussion it has been proved that students’ projects
have the potential to be enhanced and used fora prototype
of a marketable product. In order to achieve these extensions,
student projects need to follow certain standards [7], [8].
In Pakistan, the rules and goals for education programs are
stipulated by the Higher Education Commission ( HEC ). It
plays a key role in evaluating and developing curricula and, to-
gether with the universities, continuously performs curriculum
revisions in both the basic and applied sciences as well as in
the engineering disciplines. The latest curriculum revision [9]
is mainly based upon ACM/IEEE Computing Curricula 2004
[6]. The basic intention is to encourage students to reflect
upon the theories and abstract models as well as apply the
knowledge in practical situations. The capstone project is
considered as the most important course in which students
practically apply their learning experience [9].
Projects are often very different from the normal lecture
based courses.In addition to having deep knowledge of the
subject, they also require a number of generic skills such as
independent formulation of objectives, activity planning and
time management [22], [1]. According to Ahtee [4], problems
of scheduling and studying other courses in parallel with
the Project are among the major risks involved.Researchers
like Boehm and Edwards have also highlighted the need of
clear process guidelines fora successful Project [10], [7]. It
familiarizes computer science/ software engineering students
to the standards followed for a high quality software product
[11], [12]. Generally, project base courses and especially
capstone project consists of a number of development activities
for producing different documents called Deliverables and
each deliverable is a substantial piece of individual work to be
undertaken by each student. Project assessment is a requisite
to identify criteria by which students will be judged for their
effort in producing deliverables. Assessment of 3 Ps(Product,
Process and Progression in learning) of Project suggested by
Clear[13]is a formidable task. Projects are normally carried
out in teams which makes the assessment even more complex.
According to [14], faculty’s awareness of Project process and
intra-group dependencies further increases the complexity of
the assessment process.
The main purpose of this study is to provide generic support
for quality assessment of capstone projects at undergraduate
level 1. It aims to provide guidelines to teachers involved as
assessors or supervisors and also help to clarify students what
is expected from them. To develop this assessment support we
have carried out a survey at different universities in Pakistan.
Based upon this survey and document analysis, from various
universities, we have highlighted the improvement potentials
1One of the authors have previously developed rubrics and support for
quality assessment of software engineering master theses in Sweden [3]; here
we focus on the undergraduate level and the Pakistani context.
2011 Frontiers of Information Technology
978-0-7695-4625-4/11 $26.00 © 2011 IEEE
DOI 10.1109/FIT.2011.13
25
2011 Frontiers of Information Technology
978-0-7695-4625-4/11 $26.00 © 2011 IEEE
DOI 10.1109/FIT.2011.13
25
in their processes and presented systematic ways for grading
and evaluating Projects.
The next section presents the background of the study and
Section III explains the design. Section IV describes the
results and analysis and Section V highlights key challenges
and issues while Section VI presents recommendations based
upon identified challenges. Section VII concludes the study.
II. BACKGROUND
Like many other institutions of the world, it has been a
standard practice for computer science and software engi-
neering programs of Pakistani universities to incorporate a
Project in the last ( fourth ) year of their study. The quality
and output of this Project indicates the quality of program
as a whole. According to HEC ( http://www.hec.gov.pk )
there are approximately 100 universities in Pakistan, offering
undergraduate educational programs in the areas of computer
science, software engineering and information technology.
Although, HEC has developed a comprehensive curriculum
for these programs there are still variances in the structure and
management of the courses in general and Project courses in
particular. These variances in the management and assessment
not only have a direct impact on the quality of the Project, it
also affects HEC’s efforts of ensuring that all graduates have
a certain level of skills and knowledge.
Earlier studies emphasize on the importance of having
well defined process, communication with students and clear
guidelines for the assessment. According to [15] and [14]
difficulties in assessment arise due to the wide range of
topics, different availability of resources to students and most
importantly variations in the assessment criteria of the rel-
atively large academic staff involved as assessors. Divergent
approaches for assessment have been frequently reported [16],
[17], [13], [7]. Either the assessment item list is incomplete
or they miss the criteria to be used for each assessment
item [18], [19]. Assessment of professionalism emphasized by
Robert [20] instead of Progression in learning is an issue of
assessing generic skills which are really hard to judge sepa-
rately because of the lack of such information for engineering
education as reported by Feldt et al [22].
In [14], Clive presented a facilitated peer assessment ap-
proach for capstone project assessment. Clive’s approach fo-
cuses on the assessment of the later part of the Project,having
approximately 50% weight, by considering final report and
final presentation as the key assessment items. The progres-
sion of learning is evaluated in a peer assessment meeting
facilitated by a tutor at the end of the project in which
students are asked to award marks to each other. This approach
can improve students’ ability of working in a team and can
polish their decision making skills as well, however, it doesn’t
guarantee honest judgment of peers because of the honesty and
the influence of the tutor ( acting as a facilitator ). Studies in
UK [23], [24] have revealed considerable ambiguity in the
meaning, use and application of assessment criteria. It has
been reported that some of the academic staff used subjective
criteria not explicitly mentioned and others partly used the
published criteria. Another study reports that academic staff
didn’t value the guidelines given to them. It is unfair to
students who used common guidelines and yet the evaluating
staff chose their own criteria [25]. In Pakistan, there is
a lack of information of how these projects are practically
conducted and it is not clear if results from previous studies
can be applied in this context. Our experience in previously
developed rubrics and support for the quality assessment of
software engineering master theses in Sweden has been quite
successful [3], [1] and there hasn’t been any study carried
out on this scale in Pakistan. Therefore, there is a need to
investigate the guidelines about the management process and
quality assessment of capstone projects.
Hence, there is a need to investigate the need of clear guide-
lines about the management process and quality assessment
of capstone project, disseminated widely at institute and well-
know to both assessors and the students at the undergraduate
level in Pakistan.
III. METHODOLOGY
To increase the validity of results we have used variety of
research methods and combined survey based on a question-
naire with document analysis. Following we state our research
questions and describe document analysis and the web-based
questionnaire.
A. Research Questions
The following research questions have been investigated:
RQ1 What are the current practices being followed for the
assessment of capstone projects in computer science
and software engineering at different universities in
Pakistan?
RQ2 How does the practices fromRQ1vary between
universities?
RQ3 Can we formulate generic rubrics for quality assess-
ment that help minimizing the variation in quality
(from RQ2)
B. Document Analysis
Documents like Project Handbooks, Project Evaluation
Forms, Project Assessment Tools and Project Rubrics from
Nine universities SevenN ationalandT wointer national
were analyzed. Universities’ websites were the main source
of document collection and correspondence with faculty
members were carried out through emails.
C. Web-Based Questionnaire
A web-based questionnaire was administered through a web
service commonly used for questionnaires and polls ( Sur-
veyGizmo, http://www.surveygizmo.com ). Universities were
chosen from the list of degree awarding institutes mentioned
in the HEC website, which have undergraduate programs of
computer science, software engineering, and/or information
technology.We checked the universities’ websites for their
faculty members and selected them in a list if their names,
email addresses and/or contact numbers were listed. The link
2626
TABLE I
CASE INSTITUTES WITH NUMBER OF SURVEY RESPONSES.
Location (Territory/Provence) University(s) # of Responses
Balochistan A 1
Capital Territory B,C,D,E,F 7
Khyber Pakhtoonkhwa G,H,I,J 5
Punjab K,L,M,N,O,P,Q 9
Sind R,S 2
Total: 19 24
to the questionnaire and a request to participate was sent
through an email to both senior and junior faculty members
of different universities in Pakistan.Our respondents in this
survey belong to 19 different institutions, from all over the
country, covering all four provinces and the capital territory
of Pakistan. Table I shows the number of responses from
each of the participating universities. Two universities ( C and
D ) have campuses in several cities but we treated them as
separate institutions.This helped us in analyzing if different
campuses have differences in their approach. The response
rate of the survey was 40.7%. One of the main reason for
this low response rate could be the lack of any such study
carried out on this scale in Pakistan before. One of the main
issues in the data collection was to convince respondents
to answer the survey and for this reason we emailed them
several times and convinced them over phone to highlight the
importance. Even though, web-based questionnaires can have
lower response rates,however, they adequately generalize and
compare with other data collection methods [2]. One of the
authors created an initial draft of questions to be used in a
questionnaire while the other authors reviewed and updated
it slightly. The questionnaire was divided into three main
themes. The first theme focused on the context of the Project
course, the second on the documentation and strategies for
information dissemination and the third, on assessment scheme
being followed. Questions about the time to start the Project,
title of the Project course, weight of Project course in terms of
credit hours, recommended student team size for a particular
Project and the key deliverables were asked in the first theme
about Project context. In the second theme, respondents were
asked about the current knowledge of supervisors & assessors
about the Project management process, the documentation of
the process and the strategies used for process information
dissemination. Finally, the third theme contained questions
about the assessment items and percentage for each assessment
item. In addition, each theme had an open-ended question
asking about the comments and suggestions regarding the
particular theme. The respondents of the survey have varying
experience of conducting final year projects. 70% of them had
one to five years of experience whereas the remaining 30% had
ten to fifteen years or greater.
IV. RESULTS AND ANALYSIS
This section summarizes the answers to different survey
questions in different themes.
Fig. 1. Research thesis ratio in ongoing Projects.
A. Theme 1: Capstone Project Context
In most of the universities, the Project starts in the last year
(4th year). Out of 24, 21 (87.0%) reported that it starts in the
7th semester whereas 2 (8.3%) reported that it starts in the
8th last semester. Only 1 (4.2%) stated that Projects starts in
the 2nd half of the 3rd year, i.e. the 6th semester. Although,
the titles of the Project are different, most of the universities
call them Academic project. Only 1 (4.2%) has the option
to choose research thesis at undergraduate level considering
that a Project can be a software development a specific course
has 4(3+1) credit hours; this would imply that students will
have a 3 hour lecture per week along with 3 hours lab
session (considered as 1 credit hour). Minimum credit hours
for an undergraduate program in computer science/software
engineering/information technology is 130 with approximately
68.0% core and 32.0% elective courses [9].
More than 83.0% have 6-credit hours for Project, around
13.0% have 10 or more than 10 credit hours and 4.0%
have 9-credit hours. 50.0% of the respondents have Project
coordinator for Project course management, whereas ap-
prox. 42.0% mentioned only supervisors are managing this
course.8.0% mentioned: there are special interest groups to
manage Projects. Out of 24 institutes, 10 (41.7%) recom-
mended a group size of 2 to 3 students while 7 (29.2%)
recommended a group size of 1 to 2 students and 5 (20.8%)
recommend a group size of more than 3 students. Only 2
institutes (8.4%) allow students to do their Project alone.
Figure 1summarizes the answers to the question that listed the
ratio of the research thesis in ongoing Projects, keeping in view
that Project can either be a research thesis or a development
project. A research thesis ends up with a report containing
a research paper while a development project has a working
piece of software as a major output.
Figure 2 summarizes the answers to multi-select question
that listed the key deliverable of the Project course. The per-
centage value corresponding to each deliverable shows the per-
centage of the respondents that selected a particular deliverable
(possibly along with other deliverables). A final project report
is the most common deliverable with 23 (95.8%) responses.
Project proposal and an implementation with executable files
come next with 22 (91.7%) responses each. 19 (79.2%)
2727
Fig. 2. List of key Project deliverables.
respondents selected software requirement specification while
16 (66.7%) respondents selected software design document
among the available deliverables. Project poster was chosen by
8 (33.4%) respondents while only 6 (25.0%) selected software
test document as Project deliverable. Out of 24, 9(37.5%) are
also asking students for other deliverables like presentation
is considered by 5 (20.83%) and demonstration is considered
by 3 (12.5%) respondents and only 1 (4.2%) respondent is
considering research paper as Project deliverable but listed it
as optional.
B. Theme 2: Documentation and Information Dissemination
Results show that Project process and artifacts (deliverables
and their contents) are documented and well known to supervi-
sors, in most of the institutes, as reported by 75.0% (18) of the
respondents whereas 16.0% (4) suggested they are documented
but not well-known. Only 8.0% (2) states that they are not
documented but are well-known. Regular meetings between
students and their supervisor is a core medium for explaining
Project process & artifacts to students. All of the respondents
suggested they hold regular meetings with students to explain
the deliverables, process and contents of the Project and track
their performance. 29.2% of the respondents (7) are using
documents/handbooks and the same number of respondents are
conducting introductory seminars along with regular students-
supervisor meetings for the purpose. In case of industry
projects, only 4.2% (1) are involving industry representatives.
Out of 24, 15 (62.5%) are encouraging weekly student-
supervisor meeting while 7 (29.2%) are pushing for at least
one such meeting in a month. There is no such requirement
for 2 (8.4%) institutes. Some of the institutes are organizing
Open houses and Project exhibitions on the day of final
presentation and demonstration inviting people from industry.
These sessions provide a platform where students can learn
and improve their projects based upon the industrial feedback
and the people from industry have the opportunity to locate
and hire talented students. Students may also gain knowledge
of the current industry trends and can propose/select project
being accepted in the local setup. A considerable number of
students get job offers on the final day presentations in few
institutes.
Fig. 3. Roles involved in the evaluation of Projects.
C. Theme 3: Assessment
Out of 24, 18 (75%) of the respondents are doing individual
assessment and only 6 (25.0%) are evaluating the whole group
involved in a particular project. Figure 3 summarizes the
answers to multi-select question that listed the roles involved
in the evaluation of Project. The value corresponding to each
roles shows the percentage of the respondents selected that role
as involved in Project evaluation along with others. Mostly
the academic supervisor (Supervisor) is involved in Project
evaluation with 19 (79.2%) responses, along with the faculty
from the department with 16 (66.7%) responses. External
examiners are also invited by many institutes. 15 (62.5%) of
the respondents are inviting experts from industry in the final
presentation/demonstration. In case of an industry project only
9 (37.5%) of the respondents are involving industry personals
(external supervisor). Out of 24, 4 (16.7%) have formed task
forces for Project assessment. This task force may include
industry partners along with the faculty from the department
for a particular project. The task force has different names
in different institutes like project evaluation team, project
committee and project evaluation committee. Out of 24, 4
(16.7%) have formed task forces for Project assessment. This
task force may include industry partners along with faculty
from the department for a particular project. The task force has
different names in different institutes like project evaluation
team, project committee and project evaluation committee.
V. C HALLENGES AND ISSUES
Table II presents the challenges and issues identified.
The criticality level for each issue/challenge uncovered,were
judged on a scale from general, important to critical.It was
based on how frequently it was mentioned in answer to
the open-ended question asked in each theme by different
respondents and how important the researchers judged it to
be.Below is detail description of each challenge/issue.
A. Requirements on Software Process
Figure 2 shows that only 25.0% of the supervisors require
the software test document as one of the deliverables, de-
spite of the fact that majority of projects are development
projects. This supports our own experience from capstone
projects where proper testing is seldom carried out or at least
universities do not put emphasis on testing. In most of the
2828
TABLE II
CHALLENGES AND ISSUES.
Factor Challenge/Issue Severity
Process visibility Poor process visibility to all stakeholders Critical
Knowledge Difference in support and information documents within same university Critical
Support Limited guidelines for project assessment Critical
Requirements Lack of enough requirements on software process and documentation Important
Incentives Limited incentives and unclear visions General
universities, capstone project starts in the 7th semester and
they give 6 credit hours to this course. However, in one of
the university which starts project in 8th semester gave 12
credit hours. In another university, in which project starts in
the 6th semester gave 10 credit hours and their recommended
group size was one member only, whereas,in most other cases
the recommended group size was 2 to 3 students. Although
variances in this context are not substantial but as discussed
in [4], studying other courses in parallel with the Project is
one of the major risks.Starting Project in the very last (8th)
semester may either result in skipping key deliverables or may
produce a low quality project.
B. Faculty Incentives and Capstone Projects
Survey results indicate that supervisors are not often given
enough motivation to properly supervise capstone projects.
Compared to other courses there is lack of incentives and there
are hardly any clear visions for these capstone projects.
C. Process Visibility
Some of the respondents argue that there should be an inter
institute/university (at least at university level) project portal
where students can share their ideas, get help from seniors and
can view past projects (documentation only). In this way, ideas
can propagate between universities which will help create
motivation and set up a healthy competitive environment. This
will also reduce the plagiarism, where students copy projects
conducted in other institutes. Formation of heterogeneous
teams is also encouraged wherever there are multiple depart-
ments. For example students from Computer Science/Software
Engineering are engaged in developing software for a project
being conducted in Electrical or Mechatronics Engineering
department. So that interdisciplinary solutions are produced.
D. Knowledge Distribution
This section provides an overview of the variances of ap-
proaches used in capstone project documents and information
dissemination. It is interesting to note that in few cases
even different supervisors of same universities were either
not aware of the certain practices or were using different
approaches. For instance, in case of N process & artifact
were documented and well-known to the supervisors but one
of them was only conducting regular meeting whereas the
other supervisor was also using handbook to disseminate
information to students. Similar pattern is observed at different
campuses of C in Abbottabad, Lahore and Wah Cantt where
they are using handbook and regular meetings to disseminate
information, whereas only at Lahore campus they carry out
regular meetings for knowledge distribution. In case of D,
process and artifacts were documented and well-known at
campuses in Islamabad and Karachi where as in Peshawar they
are documented but not well-known. We observe variances
in the medium for information dissemination even between
the same campuses of D. One of the supervisor held regular
meetings whereas the other one also uses introductory seminar
and handbook. Quite similar is the case with I,where handbook
and regular meetings were used by one of the supervisors
whereas the other supervisor used introductory seminar and
regular meetings. Very interesting pattern is seen in case of F
where two supervisors have different views about process and
artifacts one of them chose they are documented but not well-
known and the second one chose ’not documented but well-
known’. When we cross-referenced these activities between
different institutes we observed at I, D in Peshawar campus, F
and J, Information dissemination is documented but not Well-
known, however they don’t use any handbook for the medium
of dissemination. If these activities are documented then they
must use some published material or any such document as
well.
E. Quality Assessment
Table III summarizes the results of assessment items being
considered for Project evaluation. The highest value for each
assessment item is considered to highlight the importance
of a particular item for a particular institute. To compare
questionnaire results for different items, a weighted average
for each item is calculated along with the overall average
of each item. Weighted average is calculated by dividing
total weight(summation of the percentage value given by each
respondent) by the number of responses. For example, the
weighted average for Final Demonstration is 405.08 / 19 =
21.32. The highest value of Final demonstration, Final report
and Final presentation demonstrate that they have important
roles in Project evaluation. The result shows that Project
proposal and poster are considered as the least important
assessment item with the weighted averages of 10.12, 10.42
and overall averages of 7.2 and 5.2 respectively.
Although, few institutes have defined assessment criteria
but all agreed upon having standardized assessment scheme.
The scheme must contain guideline for the criteria to be
used for Project evaluation along with the weight of different
evaluation roles. It may also contain rubrics for different
project deliverables.
2929
TABLE III
WEIGHT OF DIFFERENT ASSESSMENT ITEMS IN PROJECT EVALUATION
Assessment items Highest
value
#ofre-
sponses
Weighted
average
Overall
average
Final Demonstration 50 19 21.32 16.9
Final Report 40 19 19.74 15.6
Final Presentation 40 21 17.14 15.0
Process/Approach 35 17 16.06 11.4
Intermediate Presentations 30 12 13.75 06.9
Project proposal 20 17 10.12 07.2
Project Poster 50 12 10.42 05.2
VI. RECOMMENDATIONS BASED ON IDENTIFIED
CHALLENGES
This section discusses recommendations about how the
identified challenges can be addressed.
A. Software Process and Documentation
In general we recommend that for processes and documenta-
tion, universities should have requirements similar to industry.
Universities should collaborate with industrial partners to
achieve realistic and high level processes and documentation.
B. Clear Incentive Policy
Faculty members and other personnel involved, should be
motivated enough to improve the structure and assessment of
these projects and allocate enough time and effort during the
course. We recommend incentives and management support
for them. Ideally, special funds should be allocated. These
members should be facilitated to improve the processes
C. Process Visibility and Knowledge Distribution
It is important to have a single web portal where Project
information is disseminated to all students who are working
or planning their Projects [21]. It provides clear visibility to
all the stakeholders and a single location to share experiences.
We have successfully implemented such portals before, in a
Swedish Master Thesis course at the Blekinge Institute of
Technology. A key element in such a portal is to show detailed
examples of the whole schedule, process and intermediate as
well as final artifacts (plans as well as reports and other results)
of successful projects. Today, software industry has projects
in various domains. Universities could partly support this
by encouraging students to carry out their capstone projects
with students from other educational programs. Heterogeneous
teams with students from computer science, software engi-
neering and information technology should be allowed and
supported but also projects with students from other engineer-
ing programs. Universities need to find solutions to current
communication and scheduling problems between different
departments to make these shared projects simpler and more
common. Capstone project management process can be viewed
as set of activities in three phases; a)Pre-project b)Project and
c)Post-Project and it can be improved gradually.
a) Pre-Project Phase: This phase contains activities
which are required at the start of the project or in the early
stages. Ideally, this should be conducted systematically as it
affects the judgment regarding students’ readiness to work
in industry and the Universities’ reputation. Mary Shaw has
presented guidelines to cope with challenges which instructor
faces in balancing between technical and management topics
and defining the relation between the lecture and project
components [26]. This study recommends that preliminary
preparation, staffing, coordination between lectures and project
and clear grading policy must be considered while managing
this course. There should also be clear guidelines about
the Project management processes being followed and these
guidelines must be disseminated properly among supervisors.
Identifying stakeholders and creating Win-Win situation for
them is vital for the success of any Project [27]. Introduc-
tory seminars provide a platform for this purpose. Project
deliverables, their contents and deadlines must be defined in
the first few meetings. In case of an Industry Project, active
participation of the industrial partner is very important [28].
Assessment items, criteria and the weight for each items is an
important Pre-project activity. All the stakeholders must take
part in this activity and percentage of each assessor should be
clearly defined.
b) Project Phase: Activities performed during the devel-
opment of a Project are considered in this phase. For producing
each deliverable, defined in Pre-project phase, Plan-Do-Check-
Act cycle should be followed. Mid-Project Progress Reviews
suggested by Clear in [29]. should also be conducted. These
reviews can be very helpful in measuring the progression of
learning, the third P discussed in [13]
c) Post-Project Activities: There must be a post-mortem
analysis at the end of each capstone project based upon the
feedback of the faculty, assessors and the students involved.
Project process documents should be reviewed and updated
accordingly.
D. Support for Quality Assessment
Table IV highlights the recommended list of Project assess-
ment items along with their description and criteria. This is
based upon the Master thesis process and rubrics developed by
one of the authors at a Swedish university [3]. Furthermore,
it includes mid-project progress review and separates the final
demonstration from the final presentation. Final Presentation,
Final Demonstration and Final Report are added in the as-
sessment item list and considered as key elements in Project
assessment which is suggested by different studies [16], [17],
[13], [7], [18], [19] and supported by our survey results.
According to ACM/IEEE Computing Curricula 2004 [6] soft-
ware engineering is defined as an ’Application of systematic,
disciplined, quantifiable approach to development, operation,
and maintenance of software; that is application of engineering
to software’. We should also assess the process with which the
product is produced and not only the end result; as discussed
by Broman in [30]. Mid-Project Progress Reviews are also
the part of the assessment list to measure the progression of
3030
learning as suggested by Clear in [13], [29]. Weights of the
assessment items and criteria used within each of these items
is left to the Project stakeholders as it varies from project to
project. For example, the end product is given more weight
in industry Projects assessment and the industrial supervisor
may have the highest percentage ratio among assessors while
in case of an academic project the Process and opinion of the
academic survivor is given preference. This is in line with
the advice in [1] which recommends different support for
different types of thesis projects. Quality assessment needs
to be clear and explicitly mentioned; rubrics is a good way to
accomplish this [22], [3]. Based on the rubrics for Master
thesis projects at a Swedish university we have identified
assessment items (presented in Table IV) providing a baseline
for the development of quality assessment rubrics for capstone
projects. Each criterion is evaluated on a scale from 1 to 4,
where 1 being Minimal/no control and 4 being the Superior
command. These can further be assigned weights according to
the importance in a particular Project.
VII. CONCLUSION AND FUTURE WORK
Based on a survey of 24 responses at 19 universities in
Pakistan which are offering undergraduate programs in com-
puter science/software engineering/information technology, it
can be concluded that there are variances in the management,
assessment and support practices being followed at different
universities for capstone project courses. Even within some of
the universities there are variations. Some of the universities
have documented the process for the capstone projects but the
documents are not used as the core medium for information
dissemination; either the documents are incomplete or have
not been publicized properly among faculty or students. There
is a need for a common project portal where templates and
support material can be found but also examples of previous
projects. Overall there is a need for better guidelines regarding
the process, management strategies and assessment criteria to
fulfill the expectations of students, supervisors and industry
partners.
Based on our previous experience with creating a process
and clear assessment criteria for Master Thesis projects in
Software Engineering we propose the use of rubrics for
evaluation and assessment of capstone projects in Pakistani
universities. The rubrics specify key assessment criteria as well
as quality levels for each criteria. This makes the required
expectations more explicit and can act as a basis for more
detailed discussion and feedback for students.
We also propose that universities should view their capstone
project course as an ongoing improvement project. In contrast
to normal university courses capstone projects are adapted
every year and there is a need for continuous improvements.
In future, we aim to evaluate and discuss our rubrics within
the Pakistani university system. We also want to evaluate how
rubrics at the undergraduate level need to differ from the
master’s level.
REFERENCES
[1] M. Host, R. Feldt, and F. Luders, “Support for different roles in software
engineering master’s thesis projects, Education, IEEE Transactions on,
vol. 53, no. 2, pp. 288 –296, May 2010.
[2] S. Gosling, S. Vazire, S. Srivastava, and O. John, “Should we trust web-
based studies,” American Psychologist, vol. 59, no. 2, pp. 93–104, 2004.
[3] Claes Wohlin, Robert Feldt, Tony Gorschek and N. Laveson, “Quality
assessment rubrics and support for master theses in software engineer-
ing,” in submission, 2010.
[4] T. Ahtee and T. Poranen, “Risks in students’ software projects,” Software
Engineering Education and Training, Conference on, vol. 0, pp. 154–
157, 2009.
[5] A. T. Chamillard and K. A. Braun, “The software engineering capstone:
structure and tradeoffs,” SIGCSE Bull., vol. 34, pp. 227–231, February
2002. [Online]. Available: http://doi.acm.org/10.1145/563517.563428
[6] J. T. F. on Computing Curricula, “Software engineering 2004: Curricu-
lum guidelines for undergraduate degree program in software engineer-
ing,” IEEE and ACM, Tech. Rep, 2004.
[7] J. B. Thompson and H. M. Edwards, “A scalable approach to graduate
student projects: Hundreds with industry every year, Software Engi-
neering Education and Training, Conference on, vol. 0, pp. 158–161,
2009.
[8] S. Chenoweth, “Undergraduate software engineering students in startup
businesses,” in Proceedings of the 2008 21st Conference on Software
Engineering Education and Training. Washington, DC, USA: IEEE
Computer Society, 2008, pp. 118–125. [Online]. Available: http://portal.
acm.org/citation.cfm?id=1440459.1440824
[9] C. Division, “Curriculum of computer science,software engineering
and information technology [bs/ms], revised 2009,” Higher Education
Commission, Tech. Rep, 2009.
[10] S. Koolmanojwong and B. Boehm, “Using software project courses
to integrate education and research: An experience report,” Software
Engineering Education and Training, Conference on, vol. 0, pp. 26–33,
2009.
[11] R. Thayer and A. McGettrick, “IEEE software engineering standards: A
student’s version, in Software Engineering Education Training, 2007.
CSEET ’07. 20th Conference on, july 2007, pp. 229 –236.
[12] D. Delaney and S. Brown, “Document templates for student projects
in software engineering,” Department of Computer Science, National
University of Ireland, Maynooth, Tech. Rep, 2002.
[13] T. Clear, “Thinking issues: the three p’s of capstone project per-
formance,” SIGCSE Bull., vol. 41, pp. 69–70, June 2009. [Online].
Available: http://doi.acm.org/10.1145/1595453.1595468
[14] C. C. Rosen, “Individual assessment of group projects in software engi-
neering: A facilitated peer assessment approach,” Software Engineering
Education, Conference on, vol. 0, p. 68, 1996.
[15] J. Black, “Allocation and assessment of project work in the final year
of the engineering degree course at the university of bath, Assessment
in Higher Education, vol. 1, pp. 35–53, 1975.
[16] D. Tribe and A. Tribe, Assessing law students: Lecturers’ attitudes and
practices,” Assessment and Evaluation in Higher Education, vol. 13, pp.
195–211, 1988.
[17] M. Yorke, P. Bridges, and H. Woolf, “Mark distributions and mark-
ing practices in uk higher education some challenging issues,” Active
Learning in Higher Education, vol. 13, pp. 7–27, 2000.
[18] J. N. Koster, “Project teams and challenges in fair grading, in Proceed-
ings of the Capstone Design Conference 2010, june.
[19] B. Olsson, M. Berndtsson, B. Lundell, and J. Hansson, “Running
research-oriented final year projects for cs and is students,” SIGCSE
Bull., vol. 35, pp. 79–83, January 2003. [Online]. Available: http:
//doi.acm.org/10.1145/792548.611938
[20] R. Adams and J. Alsabbagh, “Cs capstone: Lessons from the field,” in
Proceedings of the Capstone Design Conference 2010, june.
[21] L. Wyard-Scott, An Online Tool for Capstone Design Course Com-
munication,” in Proceedings of the Capstone Design Conference 2010,
june.
[22] R. Feldt, M. H¨
ost, and F. L¨
uders, “Generic skills in software engineering
master thesis projects: Towards rubric-based evaluation, in Proceedings
of the 2009 22nd Conference on Software Engineering Education and
Training. Washington, DC, USA: IEEE Computer Society, 2009,
pp. 12–15. [Online]. Available: http://portal.acm.org/citation.cfm?id=
1524294.1524446
3131
TABLE IV
RECOMMENDED LIST OF ASSESSMENT ITEMS.
Assessment Item Description Recommended Criteria
Process To assess that student(s) have kept continuous contact during the work and have been on time both to
meetings and in sending deliverables.
Regularity, Independence
Mid-project
Progress Reviews
To assess that student (s) have completed tasks and delivered documents expected in the first half of the
course. It includes both demonstration and presentation of the work.
Subject knowledge, Organization, Con-
tents, Presentation, Completeness and
Accuracy
Final Presentation To assess problem understanding, adequate analysis, quality of the design and presentation skills. Student
(s) are required to discuss the completeness and accomplishment of the project
Supervisor, Supervisory Committee, Ex-
ternal Examiner
Final Demonstra-
tion
To assess the end product developed in terms of interfaces, coding standards, and originality of the work. It
requires students to install project and run it for real time transactions.
Task Distribution, Originality, Usability,
Help Manual, Coding Standards
Project Report To assess the structure of the project report. Student(s) are required to show planning and progress in an
organized way with emphasis on the interpretation of the information gathered during the project.
Style and Format, Contents, Language,
Structure, Completeness and Accuracy
[23] F. Webster, D. Pepper, and A. Jenkins, “Assessing the undergraduate
dissertation,” Assessment and Evaluation in Higher Education, vol. 25,
pp. 71–80, 2000.
[24] L. Hand and D. Clewes, “Marking the difference: an investigation of
the criteria used for assessing undergraduate dissertations in a business
school,” Assessment and Evaluation in Higher Education, vol. 25, pp.
5–21, 2000.
[25] J. Jawitz, S. Shay, and R. Moore, “Management and assessment of
final year projects in engineering,” International Journal of Engineering
Education, vol. 18, pp. 474–478, 2002.
[26] M. Shaw and J. E. Tomayko, “Models for undergraduate project courses
in software engineering,” in Proceedings of the SEI Conference on
Software Engineering Education. London, UK: Springer-Verlag, 1991,
pp. 33–71. [Online]. Available: http://portal.acm.org/citation.cfm?id=
648325.754779
[27] D. Port and B. Boehm, “Using a model framework in developing and
delivering a family of software engineering project courses, in Proceed-
ings of the 14th Conference on Software Engineering Education and
Training. Washington, DC, USA: IEEE Computer Society, 2001, pp.
44–. [Online]. Available: http://portal.acm.org/citation.cfm?id=794193.
794889
[28] W. W. McMillan and S. Rajaprabhakaran, “What leading practitioners
say should be emphasized in students’ software engineering projects,” in
Proceedings of the 12th Conference on Software Engineering Education
and Training. Washington, DC, USA: IEEE Computer Society,
1999, pp. 177–. [Online]. Available: http://portal.acm.org/citation.cfm?
id=520049.794268
[29] T. Clear, “Managing mid-project progress reviews: a model for formative
group assessment in capstone projects,” ACM Inroads., vol. 1, pp. 14–15,
March 2010. [Online]. Available: http://doi.acm.org/10.1145/1721933.
1721938
[30] D. Broman, “Should software engineering projects be the backbone or
the tail of computing curricula?” in Software Engineering Education
and Training (CSEE&T), 2010 23rd IEEE Conference on, 2010, pp.
153 –156.
3232
... Incorporating latest technologies, such unified management can facilitate discovering the trends and patterns related to the domain, complexity and time factor. Furthermore, one of authors was previously involved in a nation-wide study for developing rubrics and support for quality assessment of software capstone projects in Pakistan (Ahmad, Raza, & Feldt, 2011). To our best knowledge, there has not been any study carried out on this scale in Saudi Arabia. ...
... Based on the lessons learned from our previous study described in Ahmad et al. (2011), not only the formation of the questionnaire was changed but some the of individual questions were also updated (rephrased, new options were added, etc.). In addition, four new questions were added to explore current management practices in more detail. ...
... The structure of capstone projects in Saudi universities is no different than that followed by many universities across the world (Ahmad et al., 2011;Lesko, 2009;Greenberg Davis & Zilora, 2016). Students are given two courses to complete their project. ...
Article
Full-text available
Purpose Saudi universities have incorporated capstone projects in the final year of an undergraduate study. Although universities are following recommendations of the National Commission for National Commission for Academic Accreditation and Assessment (NCAAA) and Accreditation Board for Engineering and Technology (ABET), no detailed guidelines for management and assessment of capstone projects are provided by these accreditation bodies. Variation in the management and assessment practices of capstone project courses and analysis of the students' capabilities to align with industry demands, to realize Vision 2030, is challenging. This study investigates the current practices for structure definition, management and assessment criteria used for capstone project courses at undergraduate level for information technology (IT) programs at Saudi universities. Design/methodology/approach A web-based questionnaire is administered using a web service commonly used for questionnaires and polls to investigate the structure, management and assessment of capstone projects at the undergraduate level offering software engineering, computer science and information technology (SECSIT) programs. In total, 42 faculty members (with range of experience of managing/advising capstone projects from 1 to more than 10 years) from 22 Saudi universities (out of more than 30 universities offering SECSIT undergraduate programs) participated in the study. Findings The authors have identified that Saudi universities are facing challenges in the utilized process model, the distribution of work and marks, the knowledge sharing approach and the assessment scheme. To cope with these challenges, the authors recommend the use of an incremental development process, the utilization of a project-driven approach, the development of a national level digital archive and the implementation of homogeneous assessment scheme. Social implications To contribute to the national growth and to fulfill the market demand, universities are recommended to align the capstone project courses with latest technology trends. Universities must collaborate with the industry and update the structure and requirements of capstone project courses accordingly. This will further facilitate to bridge the gap between industry and academia and will develop a win–win scenario for all the stakeholders. Originality/value Although universities are committed to increase innovative capacities of their students for enabling them to contribute to economic and social growth, it is still hard to know the knowledge creation and sharing at national level. Variations in the management and assessment practices for capstone projects further intensify this challenge. Hence, there is a need of smart assessment and management of software capstone projects being developed in Saudi universities. Incorporating latest technologies, such unified management can facilitate discovering the trends and patterns related to the domain and complexity.
... Several investigations are reported in the literature to present effective Capstone Project (CP) course setups, structures, and assessment frameworks in computer science and engineering programs [1][2][3][4][5][6][7][8][9][10][11][12]. In [1], the author describes a course structure that includes a carefully designed prerequisite course on project management and scheduled milestones. ...
... Assessment rubrics were presented for software-related undergraduate capstone projects in [6]. In addition, it was recommended that capstone projects should always undergo a continuous improvement process. ...
Article
Full-text available
A capstone project is a culminating experience that entails creativity, critical thinking, and advanced problem-solving skills. To that end, capstone projects enable students to prove their abilities, demonstrate their attained skills, and carry out a significant project in their field. In Computer Science Bachelor programs, there is a strong mapping between learning outcomes of capstone projects and all student learning outcomes. This paper presents an assessment framework for capstone courses that allows for sound evaluations of the performance of students and project qualities; besides assessing the student outcomes of the program. The developed framework comprises criteria, indicators, extensive analytic rubrics, and a summative statistical formulation. The presented course and framework are supported by the results, analysis, and evaluation of a pilot study for a single institution to explore the effectiveness of the proposed tool.
... Many instructors use rubrics for grading purposes; some consider them for the purpose of improving learning. Several researchers have focused on studying how the use of rubrics to give feedback to their students can help them in their learning [1,2,3,4,5,8,12]. Some describe studies on giving students rubrics prior to assignments to improve their performance [2,5,12]. ...
... Ahmad, Raza and Feldt [1] performed a survey of 19 Pakistani universities related to assessment of capstone courses with the goal of "providing generic support for quality assessment of capstone projects at undergraduate level." Results of the survey indicated a wide variance in all aspects of the capstone courses with respect to not only assessment but also credit hours, number of semesters, group vs. individual projects, and many other features. ...
Article
Full-text available
Researchers in the past have studied the use of rubrics by instructors and the effort involved to maintain a level of fairness in grading. However, few studies have attempted an in-depth analysis of rubrics as a tool for the student to improve performance in a programming course. This paper serves to investigate the usefulness of rubrics as a programming guideline for students' assignments in Data Structures & ADTs, a second course in the computer science major. The aim is to determine whether rubrics are helpful to students in completing their assignments and achieving better grades. To this end, an analysis of the rubric scores and student surveys are conducted.
... Farrell et al. 8 describe an approach that attempts to develop a system for the fair allocation of course grades to the members of the senior project team. In grading our senior projects we use some of the ideas presented by Farrell et al. 8 , e.g., peer group assessment and evaluating meeting minutes Ahmad et al. 5 performed a study of the undergraduate software capstone project at 19 Pakistani universities and provides generic support for quality assessment of capstone projects at the undergraduate level. The study investigated the current practices followed for assessment of computer science and software engineering capstone projects and the formulation of generic rubrics for quality assessment to minimize variation in quality. ...
Conference Paper
Many undergraduate engineering programs require senior students to take capstone design courses to work on industry-sponsored projects. Such a project often involves multiple parties of people playing various roles: students as project workers, faculty members as coordinator or advisors, as well as mentors from the sponsoring industry. In order to promote student success, highly demanded are platforms that can effectively facilitate multi-party collaboration during the development process of capstone projects. CapStone is such a platform that has been developed and deployed at the Behrend College, Pennsylvania State University. In this paper, we present the design rationale of CapStone, explaining the responsibilities of different roles involved in a capstone project and how those responsibilities are meshed into one common platform such that all parties could effectively orchestrate their activities. We have also conducted a study on how well CapStone could help student users in their project development. The responses showed that the adoption of CapStone has significantly improved student success in many ways. Particularly, over 80% of the students found CapStone helpful in project planning/scheduling, reporting weekly progress to advisors, collecting feedbacks on project works, and in writing their project reports.
Article
Undergraduate program assessment is undertaken by many colleges world-wide in support of their continuous improvement processes. In addition to assuring stakeholders of program quality, assessment is required by major regional and national accrediting agencies. A critical part of the assessment process is the generation of useful data for analysis and evaluation yielding indicators for program improvement. Senior year capstone projects are a fertile source of such data. In this paper, we outline the Student Outcomes and Senior Project course of the BS-CS program at Florida International University (FIU). We describe and evaluate a methodology used to perform assessment of attainment of the BS-CS Student Outcomes using data from the Senior Project course.
Conference Paper
The objective of this work in progress is to present and to get feedback from the community on a guide for supervising Master's Thesis. The target audience are the novice supervisors of Master Thesis, who demand a practical guide about how to proceed efficiently when assigning the tasks to the student and when performing his/her own tasks. The proposed guide consists of a set of activities that are easy to implement. The proposed activities are oriented to the development of the Master Thesis itself and the application of a quality assurance process on it. In addition to some modern learning theories, the definition of the activities is supported by a review of the literature, the experience of the authors and a survey performed on supervisors and students. The theoretical basis is implicitly included in the activities so that the application of the modern learning theories is straightforward.
Conference Paper
The CS capstone (senior project) course has two primary goals: academic and practical. The academic goal is to integrate knowledge and skills developed in other courses in the context of a real-world project, while the practical goal is to carryout software development in a simulated real-world development environment. A one-semester capstone course presents a challenge to achieving these goals in such short time. This paper describes a 40-60 Teaching Approach that utilizes 40% of the course time for achieving the academic goal and the remaining 60% for achieving the practical goal. The first part focuses on the exploration, planning, design, and document generation, while the second part is dedicated for working with tools and technologies. The presented approach is the result of the author's experience in teaching and fine-tuning the capstone course since spring 2004.
Conference Paper
Full-text available
Project courses involving teams of students present a serious challenge to fair grading of individual students. There is the possibility that some students put in an enormous amount of work and others try to get a good grade with little effort. As the advising faculty member is not with the team at all times, there may be many activities by the team that go unnoticed. The adviser has the continuous challenge to find out which students perform and which students do not perform. The advisers must have a selection of instruments at hand that allow for best possible fair grading of team members.
Article
Full-text available
During the several years that we have been teaching the computer science capstone course we have learned many valuable lessons. This paper describes the pedagogy of our course along three axes: product, process, and professionalism. The paper illustrates how our course ensures that the student, by demonstrating each of these elements, will be a competent professional upon graduation. Finally, the paper concludes with other lessons learned regarding motivation, team-formation, individual student evaluations, and development methodology. Our goal is to provide ideas for others engaged in capstone education, as well as to elicit feedback from that same audience.
Article
Full-text available
Project courses can be organized efficiently in different forms involving teams of students. They provide the student with a work experience similar to industry practice. However, they present a serious challenge to fair grading of individual students. There is the possibility that some students put in an enormous amount of work and others try to get a good grade with little effort. As the advising faculty member is not with the team at all times, there may be many activities by the team that go unnoticed. The adviser has the continuous challenge to find out which students perform and which students do not perform. The advisers must have a selection of instruments at hand that allow for best possible fair grading of team members.
Article
Full-text available
This technical report outlines the contents of a minimal set of software development documents, tailored for use by students in software engineering projects, and firmly based on IEEE standards. The document set is designed to support software development activities. It provides a framework for use in undergraduate software engineering projects, both individual and team-based, that helps students to learn best practice. A supplementary report describes the content of each document in more detail.
Article
Full-text available
In this paper we discuss the difficulties of designing and running a final year project course for computer science and information systems students. In particular, we present the design of a research-oriented final year project course, where the aim is to develop the student's critical thinking and research skills. We also discuss the lessons learnt during the seven years that the course has been given.
Article
This paper reports on a study of assessment of undergraduate dissertations in the seven departments which constitute the School of Social Sciences and Law at Oxford Brookes University. Information was gathered from documented material and interviews regarding criteria used in assessment of dissertations. This was analysed to identify the range of criteria across disciplines. Criteria were also examined by administration of a questionnaire asking staff to define commonly listed criteria. Finally, a content analysis of completed assessment forms was undertaken to gauge the extent to which identified criteria were actually applied and to gain insight into those that were implicit. The study revealed considerable ambiguity as regards use, meaning and application of criteria. The paper concludes by outlining the policies and practices that are now being developed in the School in the light of these findings, and situates the study in the UK national context of concern to establish graduate threshold standards.
Article
Do we need another editorial about engaging students in learning computer science so they will stay in the field and prepare for a career or further study? We wish it were not so, but in spite of some progress, there is little evidence that our students ...
Article
This article reports some findings which have emerged from research into assessment that has been carried out over a number of years by the Student Assessment and Classification Working Group in the UK. The findings raise questions at a number of levels about assessment practice. The heterogeneity in the distribution of honours degree classes that is visible in Higher Education Statistics Agency (HESA) statistics has a parallel at the level of the module, which provokes some disquiet about the equitability of assessment across disciplinary areas. Differences between performances in coursework and examinations prompt questions about the purposes and ‘authenticity’ of assessments. Initial work on lecturers’ marking behaviour, coupled with the preceding points, suggests that there is considerable scope for professional development in the area of assessment.