Content uploaded by Ehsan Ahmad
Author content
All content in this area was uploaded by Ehsan Ahmad on May 22, 2015
Content may be subject to copyright.
Assessment and Support for Software Capstone
Projects at the Undergraduate Level:
A Survey and Rubrics
Ehsan Ahmad
School of Computer Science and Technology
Northwestern Polytechnical University
710072 Xi’an, China
Email: ehsan@mail.nwpu.edu.cn
Bilal Raza
Blekinge Institute of Technology
SE-372 25 Ronneby, Sweden
Email: bilalraza001@gmail.com
Robert Feldt
Division of Software Engineering
Chalmers University of Technology
SE-412 96 Gothenburg, Sweden
Email: robert.feldt@chalmers.se
Abstract—Software engineering and computer science students
conduct a capstone project during the final year of their degree
programs. These projects are essential in validating that students
have gained required knowledge and they can synthesize and use
that knowledge to solve real world problems. However, the exter-
nal requirements on educational programs often do not provide
detailed guidelines for how to conduct or support these capstone
projects, which may lead to variations among universities. This
paper presents the results from a survey conducted at 19 different
Pakistani universities of the current management practices and
assessment criteria used for the capstone project courses at
Undergraduate level. Based upon the results of this survey and
similar work on Master Thesis capstone projects in Sweden, we
present assessment rubrics for software-related undergraduate
capstone projects. We also present recommendations for the
continuous improvement of capstone projects.
Index Terms—Capstone Project, Thesis, Survey, Management
Practices, Assessment Rubrics, Undergraduate Level
I. INTRODUCTION
A capstone or thesis project (Project) is an important part
of most education programs especially within engineering at
the undergraduate level. The main purpose of these projects is
to encourage students to apply the knowledge acquired during
their studies, individually or in a group formed as a team [1],
[4], [5]. Students are also expected to show how good they are
in solving real world problems of limited scope, with certain
constraints for the outcome-based evaluation suggested by
ACM/IEEE Computing Curricula 2004 [6]. Through research
and discussion it has been proved that students’ projects
have the potential to be enhanced and used fora prototype
of a marketable product. In order to achieve these extensions,
student projects need to follow certain standards [7], [8].
In Pakistan, the rules and goals for education programs are
stipulated by the Higher Education Commission ( HEC ). It
plays a key role in evaluating and developing curricula and, to-
gether with the universities, continuously performs curriculum
revisions in both the basic and applied sciences as well as in
the engineering disciplines. The latest curriculum revision [9]
is mainly based upon ACM/IEEE Computing Curricula 2004
[6]. The basic intention is to encourage students to reflect
upon the theories and abstract models as well as apply the
knowledge in practical situations. The capstone project is
considered as the most important course in which students
practically apply their learning experience [9].
Projects are often very different from the normal lecture
based courses.In addition to having deep knowledge of the
subject, they also require a number of generic skills such as
independent formulation of objectives, activity planning and
time management [22], [1]. According to Ahtee [4], problems
of scheduling and studying other courses in parallel with
the Project are among the major risks involved.Researchers
like Boehm and Edwards have also highlighted the need of
clear process guidelines fora successful Project [10], [7]. It
familiarizes computer science/ software engineering students
to the standards followed for a high quality software product
[11], [12]. Generally, project base courses and especially
capstone project consists of a number of development activities
for producing different documents called Deliverables and
each deliverable is a substantial piece of individual work to be
undertaken by each student. Project assessment is a requisite
to identify criteria by which students will be judged for their
effort in producing deliverables. Assessment of 3 Ps(Product,
Process and Progression in learning) of Project suggested by
Clear[13]is a formidable task. Projects are normally carried
out in teams which makes the assessment even more complex.
According to [14], faculty’s awareness of Project process and
intra-group dependencies further increases the complexity of
the assessment process.
The main purpose of this study is to provide generic support
for quality assessment of capstone projects at undergraduate
level 1. It aims to provide guidelines to teachers involved as
assessors or supervisors and also help to clarify students what
is expected from them. To develop this assessment support we
have carried out a survey at different universities in Pakistan.
Based upon this survey and document analysis, from various
universities, we have highlighted the improvement potentials
1One of the authors have previously developed rubrics and support for
quality assessment of software engineering master theses in Sweden [3]; here
we focus on the undergraduate level and the Pakistani context.
2011 Frontiers of Information Technology
978-0-7695-4625-4/11 $26.00 © 2011 IEEE
DOI 10.1109/FIT.2011.13
25
2011 Frontiers of Information Technology
978-0-7695-4625-4/11 $26.00 © 2011 IEEE
DOI 10.1109/FIT.2011.13
25
in their processes and presented systematic ways for grading
and evaluating Projects.
The next section presents the background of the study and
Section III explains the design. Section IV describes the
results and analysis and Section V highlights key challenges
and issues while Section VI presents recommendations based
upon identified challenges. Section VII concludes the study.
II. BACKGROUND
Like many other institutions of the world, it has been a
standard practice for computer science and software engi-
neering programs of Pakistani universities to incorporate a
Project in the last ( fourth ) year of their study. The quality
and output of this Project indicates the quality of program
as a whole. According to HEC ( http://www.hec.gov.pk )
there are approximately 100 universities in Pakistan, offering
undergraduate educational programs in the areas of computer
science, software engineering and information technology.
Although, HEC has developed a comprehensive curriculum
for these programs there are still variances in the structure and
management of the courses in general and Project courses in
particular. These variances in the management and assessment
not only have a direct impact on the quality of the Project, it
also affects HEC’s efforts of ensuring that all graduates have
a certain level of skills and knowledge.
Earlier studies emphasize on the importance of having
well defined process, communication with students and clear
guidelines for the assessment. According to [15] and [14]
difficulties in assessment arise due to the wide range of
topics, different availability of resources to students and most
importantly variations in the assessment criteria of the rel-
atively large academic staff involved as assessors. Divergent
approaches for assessment have been frequently reported [16],
[17], [13], [7]. Either the assessment item list is incomplete
or they miss the criteria to be used for each assessment
item [18], [19]. Assessment of professionalism emphasized by
Robert [20] instead of Progression in learning is an issue of
assessing generic skills which are really hard to judge sepa-
rately because of the lack of such information for engineering
education as reported by Feldt et al [22].
In [14], Clive presented a facilitated peer assessment ap-
proach for capstone project assessment. Clive’s approach fo-
cuses on the assessment of the later part of the Project,having
approximately 50% weight, by considering final report and
final presentation as the key assessment items. The progres-
sion of learning is evaluated in a peer assessment meeting
facilitated by a tutor at the end of the project in which
students are asked to award marks to each other. This approach
can improve students’ ability of working in a team and can
polish their decision making skills as well, however, it doesn’t
guarantee honest judgment of peers because of the honesty and
the influence of the tutor ( acting as a facilitator ). Studies in
UK [23], [24] have revealed considerable ambiguity in the
meaning, use and application of assessment criteria. It has
been reported that some of the academic staff used subjective
criteria not explicitly mentioned and others partly used the
published criteria. Another study reports that academic staff
didn’t value the guidelines given to them. It is unfair to
students who used common guidelines and yet the evaluating
staff chose their own criteria [25]. In Pakistan, there is
a lack of information of how these projects are practically
conducted and it is not clear if results from previous studies
can be applied in this context. Our experience in previously
developed rubrics and support for the quality assessment of
software engineering master theses in Sweden has been quite
successful [3], [1] and there hasn’t been any study carried
out on this scale in Pakistan. Therefore, there is a need to
investigate the guidelines about the management process and
quality assessment of capstone projects.
Hence, there is a need to investigate the need of clear guide-
lines about the management process and quality assessment
of capstone project, disseminated widely at institute and well-
know to both assessors and the students at the undergraduate
level in Pakistan.
III. METHODOLOGY
To increase the validity of results we have used variety of
research methods and combined survey based on a question-
naire with document analysis. Following we state our research
questions and describe document analysis and the web-based
questionnaire.
A. Research Questions
The following research questions have been investigated:
RQ1 What are the current practices being followed for the
assessment of capstone projects in computer science
and software engineering at different universities in
Pakistan?
RQ2 How does the practices fromRQ1vary between
universities?
RQ3 Can we formulate generic rubrics for quality assess-
ment that help minimizing the variation in quality
(from RQ2)
B. Document Analysis
Documents like Project Handbooks, Project Evaluation
Forms, Project Assessment Tools and Project Rubrics from
Nine universities SevenN ationalandT wointer national
were analyzed. Universities’ websites were the main source
of document collection and correspondence with faculty
members were carried out through emails.
C. Web-Based Questionnaire
A web-based questionnaire was administered through a web
service commonly used for questionnaires and polls ( Sur-
veyGizmo, http://www.surveygizmo.com ). Universities were
chosen from the list of degree awarding institutes mentioned
in the HEC website, which have undergraduate programs of
computer science, software engineering, and/or information
technology.We checked the universities’ websites for their
faculty members and selected them in a list if their names,
email addresses and/or contact numbers were listed. The link
2626
TABLE I
CASE INSTITUTES WITH NUMBER OF SURVEY RESPONSES.
Location (Territory/Provence) University(s) # of Responses
Balochistan A 1
Capital Territory B,C,D,E,F 7
Khyber Pakhtoonkhwa G,H,I,J 5
Punjab K,L,M,N,O,P,Q 9
Sind R,S 2
Total: 19 24
to the questionnaire and a request to participate was sent
through an email to both senior and junior faculty members
of different universities in Pakistan.Our respondents in this
survey belong to 19 different institutions, from all over the
country, covering all four provinces and the capital territory
of Pakistan. Table I shows the number of responses from
each of the participating universities. Two universities ( C and
D ) have campuses in several cities but we treated them as
separate institutions.This helped us in analyzing if different
campuses have differences in their approach. The response
rate of the survey was 40.7%. One of the main reason for
this low response rate could be the lack of any such study
carried out on this scale in Pakistan before. One of the main
issues in the data collection was to convince respondents
to answer the survey and for this reason we emailed them
several times and convinced them over phone to highlight the
importance. Even though, web-based questionnaires can have
lower response rates,however, they adequately generalize and
compare with other data collection methods [2]. One of the
authors created an initial draft of questions to be used in a
questionnaire while the other authors reviewed and updated
it slightly. The questionnaire was divided into three main
themes. The first theme focused on the context of the Project
course, the second on the documentation and strategies for
information dissemination and the third, on assessment scheme
being followed. Questions about the time to start the Project,
title of the Project course, weight of Project course in terms of
credit hours, recommended student team size for a particular
Project and the key deliverables were asked in the first theme
about Project context. In the second theme, respondents were
asked about the current knowledge of supervisors & assessors
about the Project management process, the documentation of
the process and the strategies used for process information
dissemination. Finally, the third theme contained questions
about the assessment items and percentage for each assessment
item. In addition, each theme had an open-ended question
asking about the comments and suggestions regarding the
particular theme. The respondents of the survey have varying
experience of conducting final year projects. 70% of them had
one to five years of experience whereas the remaining 30% had
ten to fifteen years or greater.
IV. RESULTS AND ANALYSIS
This section summarizes the answers to different survey
questions in different themes.
Fig. 1. Research thesis ratio in ongoing Projects.
A. Theme 1: Capstone Project Context
In most of the universities, the Project starts in the last year
(4th year). Out of 24, 21 (87.0%) reported that it starts in the
7th semester whereas 2 (8.3%) reported that it starts in the
8th last semester. Only 1 (4.2%) stated that Projects starts in
the 2nd half of the 3rd year, i.e. the 6th semester. Although,
the titles of the Project are different, most of the universities
call them Academic project. Only 1 (4.2%) has the option
to choose research thesis at undergraduate level considering
that a Project can be a software development a specific course
has 4(3+1) credit hours; this would imply that students will
have a 3 hour lecture per week along with 3 hours lab
session (considered as 1 credit hour). Minimum credit hours
for an undergraduate program in computer science/software
engineering/information technology is 130 with approximately
68.0% core and 32.0% elective courses [9].
More than 83.0% have 6-credit hours for Project, around
13.0% have 10 or more than 10 credit hours and 4.0%
have 9-credit hours. 50.0% of the respondents have Project
coordinator for Project course management, whereas ap-
prox. 42.0% mentioned only supervisors are managing this
course.8.0% mentioned: there are special interest groups to
manage Projects. Out of 24 institutes, 10 (41.7%) recom-
mended a group size of 2 to 3 students while 7 (29.2%)
recommended a group size of 1 to 2 students and 5 (20.8%)
recommend a group size of more than 3 students. Only 2
institutes (8.4%) allow students to do their Project alone.
Figure 1summarizes the answers to the question that listed the
ratio of the research thesis in ongoing Projects, keeping in view
that Project can either be a research thesis or a development
project. A research thesis ends up with a report containing
a research paper while a development project has a working
piece of software as a major output.
Figure 2 summarizes the answers to multi-select question
that listed the key deliverable of the Project course. The per-
centage value corresponding to each deliverable shows the per-
centage of the respondents that selected a particular deliverable
(possibly along with other deliverables). A final project report
is the most common deliverable with 23 (95.8%) responses.
Project proposal and an implementation with executable files
come next with 22 (91.7%) responses each. 19 (79.2%)
2727
Fig. 2. List of key Project deliverables.
respondents selected software requirement specification while
16 (66.7%) respondents selected software design document
among the available deliverables. Project poster was chosen by
8 (33.4%) respondents while only 6 (25.0%) selected software
test document as Project deliverable. Out of 24, 9(37.5%) are
also asking students for other deliverables like presentation
is considered by 5 (20.83%) and demonstration is considered
by 3 (12.5%) respondents and only 1 (4.2%) respondent is
considering research paper as Project deliverable but listed it
as optional.
B. Theme 2: Documentation and Information Dissemination
Results show that Project process and artifacts (deliverables
and their contents) are documented and well known to supervi-
sors, in most of the institutes, as reported by 75.0% (18) of the
respondents whereas 16.0% (4) suggested they are documented
but not well-known. Only 8.0% (2) states that they are not
documented but are well-known. Regular meetings between
students and their supervisor is a core medium for explaining
Project process & artifacts to students. All of the respondents
suggested they hold regular meetings with students to explain
the deliverables, process and contents of the Project and track
their performance. 29.2% of the respondents (7) are using
documents/handbooks and the same number of respondents are
conducting introductory seminars along with regular students-
supervisor meetings for the purpose. In case of industry
projects, only 4.2% (1) are involving industry representatives.
Out of 24, 15 (62.5%) are encouraging weekly student-
supervisor meeting while 7 (29.2%) are pushing for at least
one such meeting in a month. There is no such requirement
for 2 (8.4%) institutes. Some of the institutes are organizing
Open houses and Project exhibitions on the day of final
presentation and demonstration inviting people from industry.
These sessions provide a platform where students can learn
and improve their projects based upon the industrial feedback
and the people from industry have the opportunity to locate
and hire talented students. Students may also gain knowledge
of the current industry trends and can propose/select project
being accepted in the local setup. A considerable number of
students get job offers on the final day presentations in few
institutes.
Fig. 3. Roles involved in the evaluation of Projects.
C. Theme 3: Assessment
Out of 24, 18 (75%) of the respondents are doing individual
assessment and only 6 (25.0%) are evaluating the whole group
involved in a particular project. Figure 3 summarizes the
answers to multi-select question that listed the roles involved
in the evaluation of Project. The value corresponding to each
roles shows the percentage of the respondents selected that role
as involved in Project evaluation along with others. Mostly
the academic supervisor (Supervisor) is involved in Project
evaluation with 19 (79.2%) responses, along with the faculty
from the department with 16 (66.7%) responses. External
examiners are also invited by many institutes. 15 (62.5%) of
the respondents are inviting experts from industry in the final
presentation/demonstration. In case of an industry project only
9 (37.5%) of the respondents are involving industry personals
(external supervisor). Out of 24, 4 (16.7%) have formed task
forces for Project assessment. This task force may include
industry partners along with the faculty from the department
for a particular project. The task force has different names
in different institutes like project evaluation team, project
committee and project evaluation committee. Out of 24, 4
(16.7%) have formed task forces for Project assessment. This
task force may include industry partners along with faculty
from the department for a particular project. The task force has
different names in different institutes like project evaluation
team, project committee and project evaluation committee.
V. C HALLENGES AND ISSUES
Table II presents the challenges and issues identified.
The criticality level for each issue/challenge uncovered,were
judged on a scale from general, important to critical.It was
based on how frequently it was mentioned in answer to
the open-ended question asked in each theme by different
respondents and how important the researchers judged it to
be.Below is detail description of each challenge/issue.
A. Requirements on Software Process
Figure 2 shows that only 25.0% of the supervisors require
the software test document as one of the deliverables, de-
spite of the fact that majority of projects are development
projects. This supports our own experience from capstone
projects where proper testing is seldom carried out or at least
universities do not put emphasis on testing. In most of the
2828
TABLE II
CHALLENGES AND ISSUES.
Factor Challenge/Issue Severity
Process visibility Poor process visibility to all stakeholders Critical
Knowledge Difference in support and information documents within same university Critical
Support Limited guidelines for project assessment Critical
Requirements Lack of enough requirements on software process and documentation Important
Incentives Limited incentives and unclear visions General
universities, capstone project starts in the 7th semester and
they give 6 credit hours to this course. However, in one of
the university which starts project in 8th semester gave 12
credit hours. In another university, in which project starts in
the 6th semester gave 10 credit hours and their recommended
group size was one member only, whereas,in most other cases
the recommended group size was 2 to 3 students. Although
variances in this context are not substantial but as discussed
in [4], studying other courses in parallel with the Project is
one of the major risks.Starting Project in the very last (8th)
semester may either result in skipping key deliverables or may
produce a low quality project.
B. Faculty Incentives and Capstone Projects
Survey results indicate that supervisors are not often given
enough motivation to properly supervise capstone projects.
Compared to other courses there is lack of incentives and there
are hardly any clear visions for these capstone projects.
C. Process Visibility
Some of the respondents argue that there should be an inter
institute/university (at least at university level) project portal
where students can share their ideas, get help from seniors and
can view past projects (documentation only). In this way, ideas
can propagate between universities which will help create
motivation and set up a healthy competitive environment. This
will also reduce the plagiarism, where students copy projects
conducted in other institutes. Formation of heterogeneous
teams is also encouraged wherever there are multiple depart-
ments. For example students from Computer Science/Software
Engineering are engaged in developing software for a project
being conducted in Electrical or Mechatronics Engineering
department. So that interdisciplinary solutions are produced.
D. Knowledge Distribution
This section provides an overview of the variances of ap-
proaches used in capstone project documents and information
dissemination. It is interesting to note that in few cases
even different supervisors of same universities were either
not aware of the certain practices or were using different
approaches. For instance, in case of N process & artifact
were documented and well-known to the supervisors but one
of them was only conducting regular meeting whereas the
other supervisor was also using handbook to disseminate
information to students. Similar pattern is observed at different
campuses of C in Abbottabad, Lahore and Wah Cantt where
they are using handbook and regular meetings to disseminate
information, whereas only at Lahore campus they carry out
regular meetings for knowledge distribution. In case of D,
process and artifacts were documented and well-known at
campuses in Islamabad and Karachi where as in Peshawar they
are documented but not well-known. We observe variances
in the medium for information dissemination even between
the same campuses of D. One of the supervisor held regular
meetings whereas the other one also uses introductory seminar
and handbook. Quite similar is the case with I,where handbook
and regular meetings were used by one of the supervisors
whereas the other supervisor used introductory seminar and
regular meetings. Very interesting pattern is seen in case of F
where two supervisors have different views about process and
artifacts one of them chose they are documented but not well-
known and the second one chose ’not documented but well-
known’. When we cross-referenced these activities between
different institutes we observed at I, D in Peshawar campus, F
and J, Information dissemination is documented but not Well-
known, however they don’t use any handbook for the medium
of dissemination. If these activities are documented then they
must use some published material or any such document as
well.
E. Quality Assessment
Table III summarizes the results of assessment items being
considered for Project evaluation. The highest value for each
assessment item is considered to highlight the importance
of a particular item for a particular institute. To compare
questionnaire results for different items, a weighted average
for each item is calculated along with the overall average
of each item. Weighted average is calculated by dividing
total weight(summation of the percentage value given by each
respondent) by the number of responses. For example, the
weighted average for Final Demonstration is 405.08 / 19 =
21.32. The highest value of Final demonstration, Final report
and Final presentation demonstrate that they have important
roles in Project evaluation. The result shows that Project
proposal and poster are considered as the least important
assessment item with the weighted averages of 10.12, 10.42
and overall averages of 7.2 and 5.2 respectively.
Although, few institutes have defined assessment criteria
but all agreed upon having standardized assessment scheme.
The scheme must contain guideline for the criteria to be
used for Project evaluation along with the weight of different
evaluation roles. It may also contain rubrics for different
project deliverables.
2929
TABLE III
WEIGHT OF DIFFERENT ASSESSMENT ITEMS IN PROJECT EVALUATION
Assessment items Highest
value
#ofre-
sponses
Weighted
average
Overall
average
Final Demonstration 50 19 21.32 16.9
Final Report 40 19 19.74 15.6
Final Presentation 40 21 17.14 15.0
Process/Approach 35 17 16.06 11.4
Intermediate Presentations 30 12 13.75 06.9
Project proposal 20 17 10.12 07.2
Project Poster 50 12 10.42 05.2
VI. RECOMMENDATIONS BASED ON IDENTIFIED
CHALLENGES
This section discusses recommendations about how the
identified challenges can be addressed.
A. Software Process and Documentation
In general we recommend that for processes and documenta-
tion, universities should have requirements similar to industry.
Universities should collaborate with industrial partners to
achieve realistic and high level processes and documentation.
B. Clear Incentive Policy
Faculty members and other personnel involved, should be
motivated enough to improve the structure and assessment of
these projects and allocate enough time and effort during the
course. We recommend incentives and management support
for them. Ideally, special funds should be allocated. These
members should be facilitated to improve the processes
C. Process Visibility and Knowledge Distribution
It is important to have a single web portal where Project
information is disseminated to all students who are working
or planning their Projects [21]. It provides clear visibility to
all the stakeholders and a single location to share experiences.
We have successfully implemented such portals before, in a
Swedish Master Thesis course at the Blekinge Institute of
Technology. A key element in such a portal is to show detailed
examples of the whole schedule, process and intermediate as
well as final artifacts (plans as well as reports and other results)
of successful projects. Today, software industry has projects
in various domains. Universities could partly support this
by encouraging students to carry out their capstone projects
with students from other educational programs. Heterogeneous
teams with students from computer science, software engi-
neering and information technology should be allowed and
supported but also projects with students from other engineer-
ing programs. Universities need to find solutions to current
communication and scheduling problems between different
departments to make these shared projects simpler and more
common. Capstone project management process can be viewed
as set of activities in three phases; a)Pre-project b)Project and
c)Post-Project and it can be improved gradually.
a) Pre-Project Phase: This phase contains activities
which are required at the start of the project or in the early
stages. Ideally, this should be conducted systematically as it
affects the judgment regarding students’ readiness to work
in industry and the Universities’ reputation. Mary Shaw has
presented guidelines to cope with challenges which instructor
faces in balancing between technical and management topics
and defining the relation between the lecture and project
components [26]. This study recommends that preliminary
preparation, staffing, coordination between lectures and project
and clear grading policy must be considered while managing
this course. There should also be clear guidelines about
the Project management processes being followed and these
guidelines must be disseminated properly among supervisors.
Identifying stakeholders and creating Win-Win situation for
them is vital for the success of any Project [27]. Introduc-
tory seminars provide a platform for this purpose. Project
deliverables, their contents and deadlines must be defined in
the first few meetings. In case of an Industry Project, active
participation of the industrial partner is very important [28].
Assessment items, criteria and the weight for each items is an
important Pre-project activity. All the stakeholders must take
part in this activity and percentage of each assessor should be
clearly defined.
b) Project Phase: Activities performed during the devel-
opment of a Project are considered in this phase. For producing
each deliverable, defined in Pre-project phase, Plan-Do-Check-
Act cycle should be followed. Mid-Project Progress Reviews
suggested by Clear in [29]. should also be conducted. These
reviews can be very helpful in measuring the progression of
learning, the third P discussed in [13]
c) Post-Project Activities: There must be a post-mortem
analysis at the end of each capstone project based upon the
feedback of the faculty, assessors and the students involved.
Project process documents should be reviewed and updated
accordingly.
D. Support for Quality Assessment
Table IV highlights the recommended list of Project assess-
ment items along with their description and criteria. This is
based upon the Master thesis process and rubrics developed by
one of the authors at a Swedish university [3]. Furthermore,
it includes mid-project progress review and separates the final
demonstration from the final presentation. Final Presentation,
Final Demonstration and Final Report are added in the as-
sessment item list and considered as key elements in Project
assessment which is suggested by different studies [16], [17],
[13], [7], [18], [19] and supported by our survey results.
According to ACM/IEEE Computing Curricula 2004 [6] soft-
ware engineering is defined as an ’Application of systematic,
disciplined, quantifiable approach to development, operation,
and maintenance of software; that is application of engineering
to software’. We should also assess the process with which the
product is produced and not only the end result; as discussed
by Broman in [30]. Mid-Project Progress Reviews are also
the part of the assessment list to measure the progression of
3030
learning as suggested by Clear in [13], [29]. Weights of the
assessment items and criteria used within each of these items
is left to the Project stakeholders as it varies from project to
project. For example, the end product is given more weight
in industry Projects assessment and the industrial supervisor
may have the highest percentage ratio among assessors while
in case of an academic project the Process and opinion of the
academic survivor is given preference. This is in line with
the advice in [1] which recommends different support for
different types of thesis projects. Quality assessment needs
to be clear and explicitly mentioned; rubrics is a good way to
accomplish this [22], [3]. Based on the rubrics for Master
thesis projects at a Swedish university we have identified
assessment items (presented in Table IV) providing a baseline
for the development of quality assessment rubrics for capstone
projects. Each criterion is evaluated on a scale from 1 to 4,
where 1 being Minimal/no control and 4 being the Superior
command. These can further be assigned weights according to
the importance in a particular Project.
VII. CONCLUSION AND FUTURE WORK
Based on a survey of 24 responses at 19 universities in
Pakistan which are offering undergraduate programs in com-
puter science/software engineering/information technology, it
can be concluded that there are variances in the management,
assessment and support practices being followed at different
universities for capstone project courses. Even within some of
the universities there are variations. Some of the universities
have documented the process for the capstone projects but the
documents are not used as the core medium for information
dissemination; either the documents are incomplete or have
not been publicized properly among faculty or students. There
is a need for a common project portal where templates and
support material can be found but also examples of previous
projects. Overall there is a need for better guidelines regarding
the process, management strategies and assessment criteria to
fulfill the expectations of students, supervisors and industry
partners.
Based on our previous experience with creating a process
and clear assessment criteria for Master Thesis projects in
Software Engineering we propose the use of rubrics for
evaluation and assessment of capstone projects in Pakistani
universities. The rubrics specify key assessment criteria as well
as quality levels for each criteria. This makes the required
expectations more explicit and can act as a basis for more
detailed discussion and feedback for students.
We also propose that universities should view their capstone
project course as an ongoing improvement project. In contrast
to normal university courses capstone projects are adapted
every year and there is a need for continuous improvements.
In future, we aim to evaluate and discuss our rubrics within
the Pakistani university system. We also want to evaluate how
rubrics at the undergraduate level need to differ from the
master’s level.
REFERENCES
[1] M. Host, R. Feldt, and F. Luders, “Support for different roles in software
engineering master’s thesis projects,” Education, IEEE Transactions on,
vol. 53, no. 2, pp. 288 –296, May 2010.
[2] S. Gosling, S. Vazire, S. Srivastava, and O. John, “Should we trust web-
based studies,” American Psychologist, vol. 59, no. 2, pp. 93–104, 2004.
[3] Claes Wohlin, Robert Feldt, Tony Gorschek and N. Laveson, “Quality
assessment rubrics and support for master theses in software engineer-
ing,” in submission, 2010.
[4] T. Ahtee and T. Poranen, “Risks in students’ software projects,” Software
Engineering Education and Training, Conference on, vol. 0, pp. 154–
157, 2009.
[5] A. T. Chamillard and K. A. Braun, “The software engineering capstone:
structure and tradeoffs,” SIGCSE Bull., vol. 34, pp. 227–231, February
2002. [Online]. Available: http://doi.acm.org/10.1145/563517.563428
[6] J. T. F. on Computing Curricula, “Software engineering 2004: Curricu-
lum guidelines for undergraduate degree program in software engineer-
ing,” IEEE and ACM, Tech. Rep, 2004.
[7] J. B. Thompson and H. M. Edwards, “A scalable approach to graduate
student projects: Hundreds with industry every year,” Software Engi-
neering Education and Training, Conference on, vol. 0, pp. 158–161,
2009.
[8] S. Chenoweth, “Undergraduate software engineering students in startup
businesses,” in Proceedings of the 2008 21st Conference on Software
Engineering Education and Training. Washington, DC, USA: IEEE
Computer Society, 2008, pp. 118–125. [Online]. Available: http://portal.
acm.org/citation.cfm?id=1440459.1440824
[9] C. Division, “Curriculum of computer science,software engineering
and information technology [bs/ms], revised 2009,” Higher Education
Commission, Tech. Rep, 2009.
[10] S. Koolmanojwong and B. Boehm, “Using software project courses
to integrate education and research: An experience report,” Software
Engineering Education and Training, Conference on, vol. 0, pp. 26–33,
2009.
[11] R. Thayer and A. McGettrick, “IEEE software engineering standards: A
student’s version,” in Software Engineering Education Training, 2007.
CSEET ’07. 20th Conference on, july 2007, pp. 229 –236.
[12] D. Delaney and S. Brown, “Document templates for student projects
in software engineering,” Department of Computer Science, National
University of Ireland, Maynooth, Tech. Rep, 2002.
[13] T. Clear, “Thinking issues: the three p’s of capstone project per-
formance,” SIGCSE Bull., vol. 41, pp. 69–70, June 2009. [Online].
Available: http://doi.acm.org/10.1145/1595453.1595468
[14] C. C. Rosen, “Individual assessment of group projects in software engi-
neering: A facilitated peer assessment approach,” Software Engineering
Education, Conference on, vol. 0, p. 68, 1996.
[15] J. Black, “Allocation and assessment of project work in the final year
of the engineering degree course at the university of bath,” Assessment
in Higher Education, vol. 1, pp. 35–53, 1975.
[16] D. Tribe and A. Tribe, “Assessing law students: Lecturers’ attitudes and
practices,” Assessment and Evaluation in Higher Education, vol. 13, pp.
195–211, 1988.
[17] M. Yorke, P. Bridges, and H. Woolf, “Mark distributions and mark-
ing practices in uk higher education some challenging issues,” Active
Learning in Higher Education, vol. 13, pp. 7–27, 2000.
[18] J. N. Koster, “Project teams and challenges in fair grading,” in Proceed-
ings of the Capstone Design Conference 2010, june.
[19] B. Olsson, M. Berndtsson, B. Lundell, and J. Hansson, “Running
research-oriented final year projects for cs and is students,” SIGCSE
Bull., vol. 35, pp. 79–83, January 2003. [Online]. Available: http:
//doi.acm.org/10.1145/792548.611938
[20] R. Adams and J. Alsabbagh, “Cs capstone: Lessons from the field,” in
Proceedings of the Capstone Design Conference 2010, june.
[21] L. Wyard-Scott, “An Online Tool for Capstone Design Course Com-
munication,” in Proceedings of the Capstone Design Conference 2010,
june.
[22] R. Feldt, M. H¨
ost, and F. L¨
uders, “Generic skills in software engineering
master thesis projects: Towards rubric-based evaluation,” in Proceedings
of the 2009 22nd Conference on Software Engineering Education and
Training. Washington, DC, USA: IEEE Computer Society, 2009,
pp. 12–15. [Online]. Available: http://portal.acm.org/citation.cfm?id=
1524294.1524446
3131
TABLE IV
RECOMMENDED LIST OF ASSESSMENT ITEMS.
Assessment Item Description Recommended Criteria
Process To assess that student(s) have kept continuous contact during the work and have been on time both to
meetings and in sending deliverables.
Regularity, Independence
Mid-project
Progress Reviews
To assess that student (s) have completed tasks and delivered documents expected in the first half of the
course. It includes both demonstration and presentation of the work.
Subject knowledge, Organization, Con-
tents, Presentation, Completeness and
Accuracy
Final Presentation To assess problem understanding, adequate analysis, quality of the design and presentation skills. Student
(s) are required to discuss the completeness and accomplishment of the project
Supervisor, Supervisory Committee, Ex-
ternal Examiner
Final Demonstra-
tion
To assess the end product developed in terms of interfaces, coding standards, and originality of the work. It
requires students to install project and run it for real time transactions.
Task Distribution, Originality, Usability,
Help Manual, Coding Standards
Project Report To assess the structure of the project report. Student(s) are required to show planning and progress in an
organized way with emphasis on the interpretation of the information gathered during the project.
Style and Format, Contents, Language,
Structure, Completeness and Accuracy
[23] F. Webster, D. Pepper, and A. Jenkins, “Assessing the undergraduate
dissertation,” Assessment and Evaluation in Higher Education, vol. 25,
pp. 71–80, 2000.
[24] L. Hand and D. Clewes, “Marking the difference: an investigation of
the criteria used for assessing undergraduate dissertations in a business
school,” Assessment and Evaluation in Higher Education, vol. 25, pp.
5–21, 2000.
[25] J. Jawitz, S. Shay, and R. Moore, “Management and assessment of
final year projects in engineering,” International Journal of Engineering
Education, vol. 18, pp. 474–478, 2002.
[26] M. Shaw and J. E. Tomayko, “Models for undergraduate project courses
in software engineering,” in Proceedings of the SEI Conference on
Software Engineering Education. London, UK: Springer-Verlag, 1991,
pp. 33–71. [Online]. Available: http://portal.acm.org/citation.cfm?id=
648325.754779
[27] D. Port and B. Boehm, “Using a model framework in developing and
delivering a family of software engineering project courses,” in Proceed-
ings of the 14th Conference on Software Engineering Education and
Training. Washington, DC, USA: IEEE Computer Society, 2001, pp.
44–. [Online]. Available: http://portal.acm.org/citation.cfm?id=794193.
794889
[28] W. W. McMillan and S. Rajaprabhakaran, “What leading practitioners
say should be emphasized in students’ software engineering projects,” in
Proceedings of the 12th Conference on Software Engineering Education
and Training. Washington, DC, USA: IEEE Computer Society,
1999, pp. 177–. [Online]. Available: http://portal.acm.org/citation.cfm?
id=520049.794268
[29] T. Clear, “Managing mid-project progress reviews: a model for formative
group assessment in capstone projects,” ACM Inroads., vol. 1, pp. 14–15,
March 2010. [Online]. Available: http://doi.acm.org/10.1145/1721933.
1721938
[30] D. Broman, “Should software engineering projects be the backbone or
the tail of computing curricula?” in Software Engineering Education
and Training (CSEE&T), 2010 23rd IEEE Conference on, 2010, pp.
153 –156.
3232