Developing a combined framework for evaluating IT projects based on IT-BSC and COBIT

Abstract

IT projects have special features like high risks, intangible outcomes, hidden benefits and high costs that make the use of traditional evaluation techniques very difficult and the outcomes unreliable, therefore information technology (IT) investments should be carefully controlled and evaluated. Large efforts have been done to provide appropriate methods for evaluating IT projects. In this paper, traditional methods of measuring projects are reviewed and advantages and disadvantaged of them in measuring IT projects are specified. Then new techniques that are specialized in measuring IT projects are introduced and IT-BSC as an appropriate method with available information is selected. As COBIT framework is defined for IT projects, from the combination of COBIT and IT-BSC, a framework for evaluating IT projects is proposed. Besides, the proposed framework is validated by experts' opinion with questionnaire and appropriate hypothesis test. Finally, an implemented project in IT area is selected as a case study and is evaluated by measures of the proposed framework. The results show that this framework can be very effective in evaluating IT projects.

Full text

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
Developing a Combined Framework for Evaluating IT Projects based on
IT-BSC and COBIT
1 Nasrin Dastranj Mamaghani, 2 Reza Samizadeh, 3 Fatemeh Saghafi
1, Corresponding Author Iran Telecommunication Research Center, dastranj_n@itrc.ac.ir
2, Department of Science and Technology, Alzahra University, rsamizadeh@alzahra.ac.ir
3, Iran Telecommunication Research Center, saghafi@itrc.ac.ir
doi : 10.4156/jdcta.vol5.issue5.2
ABSTRACT
IT projects have special features like high risks, intangible outcomes, hidden benefits and high costs
that make the use of traditional evaluation techniques very difficult and the outcomes unreliable,
therefore information technology (IT) investments should be carefully controlled and evaluated. Large
efforts have been done to provide appropriate methods for evaluating IT projects. In this paper,
traditional methods of measuring projects are reviewed and advantages and disadvantaged of them in
measuring IT projects are specified. Then new techniques that are specialized in measuring IT projects
are introduced and IT-BSC as an appropriate method with available information is selected. As COBIT
framework is defined for IT projects, from the combination of COBIT and IT-BSC, a framework for
evaluating IT projects is proposed. Besides, the proposed framework is validated by experts’ opinion
with questionnaire and appropriate hypothesis test. Finally, an implemented project in IT area is
selected as a case study and is evaluated by measures of the proposed framework. The results show
that this framework can be very effective in evaluating IT projects.
Keywords: IT-BSC, COBIT, Delphi Technique, IT Project, Project Management
1. Introduction
In today competitive business environment, there is strong need to control costs and returns on
investment along with lower risk of capital investment. Understanding the potential impact of IT on
strategic power of organizations and increasing payments of IT projects, has made the evaluation and
control of capital investments in IT projects remarkably important and critical [1,3]. According to
studies, only 18% of organizations use proper methods of calculating benefits of capital investment in IT
[1] and at least 22% of capital investment in IT is regarded as waste. Also, 34 to 40 percent of IT
projects have no benefit to be obtained [4.5]. Reasons for this failure are complex. Technical factors, and
in addition to that, environmental, organizational, managerial and human resources contribute to this
failure. In some studies [1, 2, 3] lack of correct evaluation and control of IT projects is the main factor of
failure in this projects. In this paper, different methods of evaluating capital investment of IT projects are
classified. In Section 2 the traditional assessment methods of projects are investigated and their
advantages and disadvantages for evaluating IT project are expressed. Section 3 introduces new ways to
evaluate IT projects. They are specific methods of IT projects assessment. Among these methods IT-
BSC1 and COBIT2 are selected as they are specified for IT projects and their information can be
available. With merging these two frameworks, a framework for evaluating organizational IT projects is
presented in Section 4. Finally, a sample project is evaluated using the proposed framework.
2. Traditional methods of project evaluaiont
Many studies investigated the benefits and drawbacks of traditional methods. Main goal of the
organization is to increase profits and wealth of project stakeholders. To calculate profit and loss of a
project, operational methods of assessment are needed like traditional methods. Also these methods are
known and understandable and are based on accepted principles [9,13,14]. But there are a lot of
1Balanced Scorecard
2 Control Objectives for Information and related Technology
- 10 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
disadvantages too. Disadvantages of using traditional methods in evaluating IT projects can be
categorized in two groups: conceptual reasons and operational reasons.
Conceptual reasons: When a company invests in the IT project, five different partners will be
involved: Main Organization (project owner), users, project teams, sponsors and shareholders. Each of
the partners follows different objectives to meet outputs of the project. If mentioned methods are used,
only the goal of project manager that is the implementation of project in time, budget and provided
description will satisfy [15,16,17,18]. Also these methods are naturally conservative and suitable for
low-risk projects and they are considered to have quick cost returns [2, 7, 10, 19].
Operational reasons: Benefits of IT projects are classified in three categories: tangible benefits that
are quantitative, intangible benefits that are qualitative and difficult to measure, hidden benefits that may
be considered very important or to be ignored [8]. For IT projects, proportion of advantages of tangible
benefits to intangible and hidden benefits are less. Because IT investments usually have supporting state
and their benefits are hidden. Traditional methods only measure the tangible benefits. Although the costs
are more measurable than benefits, significant amounts of capital cost of investment in IT projects are
intangible or hidden [14, 19]. Approximately 40% of the costs spent on IT projects do not be considered
in traditional budget estimates. 65% of IT managers are not able to calculate the full cost of the project
through traditional methods [19, 22]. There are many risks in IT capital investments, Such as risk
assessment, technical risk, project risk, operational risk, internal political risks, and external
environmental risks. By using traditional methods, the risks must be reflected in the discount rate [14,
20]. But only 7% of organizations consider discount rate in IT projects [9]. Even when managers are
interested to consider discount rate, the level of risk is usually calculated experimentally or based on
numbers [7]. Not considering Discount rate in projects lead to high-risk projects and highly considering
it result to conservative projects.
3. New methods of evaluating IT projects
As it is shown in Table2, many studies have acknowledged that traditional methods of evaluating IT
projects are not appropriate. Consequently lots of efforts have been done to replace these methods.
These efforts take two different approaches. The first approach is based on traditional methods and
decides to improve and resolve the major criticisms of traditional methods. The second approach does
not accept the traditional methods and use new methods for evaluating IT projects. Many new methods
for evaluating IT projects are still in the conceptual phase and are implemented limitedly [14]. These
methods introduced here generally and they are categorized base on the solutions they provide.
Strategic approach: the strategic approach was first introduced by Porter. He proposed strategic
dimensions to create competitive advantages clearly. Capital investment of IT projects should create
competitive advantages for organizations [23, 24]. The main tool for understanding the role of
technology in competitive advantage is Porter's value chain that makes alignment between capital
investment of IT projects and company goals. There is no explicit attention to financial analysis in this
approach [24]. This method provides the guidelines for selecting different investments, but it is not
appropriate when the organization wants to choose between two incompatible projects.
Information economics: Parker and Benson developed the information economy framework and
published it in 1987 [25, 26]. Information economy benefits from the allocation of point-rating scores
process for evaluating the benefits of capital investment and strategic communications with IT.
Information economy introduces a range of procedures which must be performed in the specified order:
1) forming a committee of managers, 2) consensus on intangible results, 3) quantifying importance of
benefits and risks in relative scale, 4) Estimating the probability scale in zero or 5, 5) multiplying each
estimate in weight and probability, 6) calculating total value, selecting the best alternative which take the
largest sum [7, 26]. Drawback of this method is that it needs public consensus. Other various methods
have been proposed in the context of Information economy, one of which is Oracle CB – 90 that is a
software model for IT managers that have developed by Oracle to do step by step process to make
decision of capital investment in IT projects. This framework leads to a decision matrix.
Options model: options model not only are used for improving traditional methods but some new
evaluation method have been developed based on theses models. Origin of such evaluating methods is in
the evaluating of complex financial transactions such as buying and selling stocks and currency
- 11 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
transactions. These models not only pay attention to financial tracking, storage and analysis of processes
but also analyses what occurs in the case of using other options. This model changes dynamically its
structure and coefficients to improve the chances in the decision [27]. Equation of Cox-Rubenstein
model is an example of options model that is used as a base for creating new Feasibility assessment
method [7, 27]. Of course there are some doubts on this method that can be referred as follow: 1– quality
of outputs not only depend on the model but depends on the inputs. Low quality of data and conflicts
will affect outputs, 2 - Suggestions for capital investment in IT projects happens less in comparison with
the number of financial transactions, 3 - Most of these methods are complex and difficult to understand
and utilize. Before recruitment of these methods, suggestions must be approved for the applicability of
these methods in the management of portfolio investment [7]. Multilayered evaluation process: This
method uses assessment methods that more or less are hierarchical. Usually a combination of traditional
methods and strategic approach is used. it is believed that the evaluation of capital investment in IT
Projects will be effective only when the evaluation process is included in the higher levels of business
processes [21, 28] and so traditional methods are not completely put away because they can create many
benefits [6, 28]. This method usually leads to a process of two or multilayered process. In the first stage,
all capital investments that are not related to business goals of organization are eliminated. In the second
stage, selection on the remaining projects is done based on calculations obtained form modified
traditional methods. For example Earl [6] has broken the second stage to three sub stages: 1- use NPV
on the basis of tangible costs and benefits (quantitative), 2- list the intangible costs and benefits
(qualitative), 3 - analyze risks and uncertainties. Finally managers must make their decisions based on
the information obtained.
IT-BSC approach: IT projects include the production of IT application programs, programs under
Internet, work with the data bases and searching the data and so on. As mentioned in previous sections,
these projects have specific characteristics that the most important of them are intangibility and inability
to calculate the full cost of the project. Therefore, using traditional criteria that evaluate solely based on
financial criteria, is insufficient. So BSC for IT projects is specialized as IT-BSC. Use of IT-BSC from
its early concept began since 1985 [36]. Gold in 1992 and Vilkoks in 1995 made a conceptual analysis
of it that is more developed by Grembergan in 1997 and later by van Bergen and Tymerman in 1998
[37]. Also in 2005, designing performance management system aimed at monitoring and promoting the
use of IT through IT-BSC was investigated [38]. Other similar research with the aim of using the BSC in
the evaluation of organizational system performance has been done [39]. In that study all aspects that the
researchers of IT has added to general BSC model to evaluate the performance, have been investigated
and the framework for IT-BSC is presented based on measurements and strategies used in them.
Researches have shown that aspects of BSC should be changed as follows in the field of IT [39, 40, 41].
1- Company contribution: means that how much value the capital investment in IT projects creates to
business and organization.
2 - Users and customers: the evaluation of stakeholders from IT facilities which are created in the project.
3- Operational excellence: the promotion of IT products and services which are created by the assessed
project, how much depend on processes that develop and deliver software applications and information
systems.
4 - Future orientation: how much opportunities from the perspective of human resources and technology
have been created to respond to future needs of organization in the evaluated project.
Each of these aspects must be translated to indicators and measurable criteria related to the leading
project to evaluate the current state. This evaluation should be periodically repeated and objectives
already determined by using information of the best models, should be compared [37]. Other
mechanisms of communication between IT and business is development of waterfall BSC [42] In a
study in 1999 by Van der. Zee, designing "operational IT-BSC" and "developmental IT-BSC” for the
development of waterfall IT-BSC has been proposed. The following figure shows the model [43].
- 12 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
Figure 1. development of waterfall IT-BSC
This model will align business strategies with IT strategies. This model show how IT creates value
through business [44]. Several studies have used BSC to evaluate projects in the field of e-commerce
[34], R & D projects [33], IT projects [29] and ICT projects [35].
COBIT Framework: COBIT framework is designed as a tool for managers in the field of IT
governance to understand and consider risk management and benefits from IT and related technologies.
This model is independent from technical IT infrastructure that is used in each organization. It’s an open
standard for IT control that has developed by the "Institute of IT governance". This framework provides
substantial approach for implementing leading projects related to IT in a controlled environment [42].
COBIT framework was presented first by ISACA and ITGI in April 1996. Its goal is providing best IT
functions for IT staff, auditors and users. Then other versions are presented by development of
managerial guidelines. In May 2007, Version 4.1 has been introduced that in relation to previous
versions have more simple definition of goals and includes waterfall development of processes and
communications between business, IT goals, and IT processes [43]. Many organizations have used
COBIT to manage or control IT. This framework is a set of 138 control objectives, a set of guidelines for
auditors and collection of implementing tools [42].
IT Sources in the COBIT framework include:
• Application programs; include automated systems and procedures for information processing
• Information; contains organization’s data such as inputs, processing and output of mechanized
systems that is used by business
• Infrastructure; including technology and equipment (such as hardware, operating system, network,
multimedia and supporting environment) used for the empowerment of processing in application
programs
• People; include staff required for planning, organizing, acquisition, implementation, presentation,
support, monitoring and assessment systems and internal, outsourcing or contract information services.
Department of IT in organizations should have clear vision of their IT processes for achieving their
objectives. These processes carried out business application programs by using people skills and
technology infrastructure, in order to strengthen the organization’s information. These resources along
with IT processes constitute, IT Architecture of organization.
COBIT conceptual framework includes three aspects of “IT processes”, “IT criteria”, and “IT
resources”. Among those in the IT processes aspect, 34 processes in the management of IT in four main
areas, including “planning and organizing”, “Acquisition & Implementation”, “delivery and support”
and “monitoring and evaluation” has been determined. Figure 2 shows the relation of these four areas. In
the second aspect or aspects of information criteria, security, quality and control needs are responded by
proposing seven criteria that can be defined as what the organization wants of IT in general. These
criteria include: effectiveness, efficiency, confidentiality, integrity, availability, compliance and
reliability. In the third aspect or aspects of IT resources, five different types of sources for IT including
data, software systems, technology, facilities and human resources has been defined [44].
4. Research Methodology
To extract goals and measures in each aspect of proposed framework, goals and metrics in the first
level of COBIT IT processes has been used and they are allocated to different aspects of the framework.
- 13 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
For allocating goals to each aspect, a complete list of goals in the COBIT IT processes (including 27
goals) were given to IT experts and managers to be selected and prioritized separately for each aspect.
As in proposed framework, there are 4 aspects, SO 4 questionnaires were designed and experts’ opinions
were analyzed in each one. In the designed questionnaires, to determine importance of each objective in
each aspect, different questions were asked which could be answered in likert scale with 5 point:
strongly disagree (1), disagree (3), neither agree nor disagree (5), agree (7) and strongly agree (9). The
method of model evaluation was based on survey-description research method and by appropriate
statistical tests, questionnaires were analyzed.
One of the most popular ways of identification of the data statistical distribution is one-sample
Kolmogorov-Smirnov test. The Kolmogorov-Smirnov test compares the observed cumulative
distribution function for a variable with a specified theoretical distribution, which may be normal,
uniform, Poisson, or exponential [31]. According to our test results, the p-value of all questions was less
than 0.05, which showed that the distribution of them was not normal. So, a statistical non-parametric
test should be used. In this way, the binomial test is used as a non-parametric test in this paper to
determine the importance of the objectives from the view of experts. Binomial test for each question is
conducted. In this stage 7 first priority objectives in each aspect have been selected as the main goals in
each aspect because when organization can plan achieve its goal that have limited and distinguished
objectives (5 to 7) to be able to extract the key success factors among its goals [45][49]. Then by using
Friedman test, objectives were prioritized. To evaluate reliability of the questionnaires, Cronbach's Alfa
method was used.
5. Presenting integrated IT-BSC and COBIT framework to evaluate IT projects
To provide integrated IT-BSC and COBIT framework, 4 aspects including “Company contribution”,
“users and customers”, “operational excellence” and “Future orientation” which were referred in various
studies [39, 40, 41] was considered for IT-BSC. To extract goals and measures in each aspect, goals and
metrics in the first level of COBIT IT processes has been used and allocated to different aspects of the
framework. For allocating goals to each aspect, a complete list of all goals in the COBIT IT processes
(including 27 goals) are given to IT experts and managers to be prioritized separately for each aspect.
Questionnaires distributed among 40 experts and 36 of them were received. Among respondents, 20
were between 20 and 30 years old, 13 people were between 30 and 40 and 3 people were over 40 years
old and the number of persons having PhD were 10, 6 persons were PhD students, 10 of them had M.S.
and 10 were B.S.. Questionnaire distributed between people who had expertise in related field, among
them 10 had experience of 5 years, 14 had experience of 12 years and 4 hadn’t determined their
experience. By using binomial test, 7 first priority objectives in each aspect have been selected as the
main goals and others were rejected because when organization can plan to achieve its goal that have
limited and distinguished objectives (5 to 7) to be able to extract the key success factors among its goals
[45].
Here the results of aspect of “customers and users” questionnaire analysis are shown. The value of
Coronbach’s Alfa for this questionnaire was 0.874 which is more than 0.7 which shows high reliability
of results of the questionnaire. Binomial test for each question is conducted, thus one of them is
presented in the following as an example:
H0: “Respond to business requirements align with business strategy” isn’t important as an objective of
company contribution.
H1: “Respond to business requirements align with business strategy” is important as an objective of
company contribution.
The result of binomial test for the “customers and users” aspect is shown in Table1:
- 14 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
Table 1. Binomial test for testing hypothesis for importance of each objective of “customers and users” aspect
Code
in COBIT objective Percent
of
consensus
Approved or
non approved
of H1
hypothesis
Asymp. Sig.
(p-value)
PO1PO2،Po،
PO10،AI1،AI6
،AI7،DS1،DS3
،ME1
Respond to business requirements align with
business strategy 50 Isn’t Approved .142
Po2،AI4،AI7 ensure integrating of application programs
with business processes 60 Isn’t Approved .196
PO5،DS6 IT share in business profitability 60 Isn’t Approved .021
PO6،AI7،DS5 ensure automating of business transactions
and reliability of information transfer 70 Isn’t Approved .075
AI1،AI2،AI6
Ensure cause and effect relation of control
and functional requirements with effective
automated solutions
60 Isn’t Approved .095
PO6،PO9 Clarify the effects business risks in IT
objectives and resources 70 Isn’t Approved .125
PO5،PO6،DS1
،DS2،DS6،
ME1،ME4
Clarify expenses, benefits, strategies, policies
and level of IT services 100 Approved .000
PO1،Po4،Po10
،ME1،ME4
Respond to employers governance
requirements 30 Isn’t Approved .325
PO5،DS6،ME1
،ME4
Ensure continuous improvement and
readiness for change in future 50 Isn’t Approved .025
PO2،DS11 Optimize using of information 90 Approved .008
Po2،Po4،PO7،
AI3
Create agility in IT processes 70 Isn’t Approved .065
PO3،AI3،DS3،
DS7،DS9
Optimize IT infrastructures, resources and
capabilities 70 Isn’t Approved .045
PO3،AI2،AI3،
AI5
Create and maintain standard and integrated
application systems 60 Isn’t Approved .125
PO6،AI4،AI7،
DS7،DS8
Ensure appropriate use and performance of
technology and application’s solutions 80 Approved .008
PO6،DS5،
DS11،DS12
Ensure confidentiality of critical information 90 Approved .004
PO7،AI5 Create and maintain IT skills that are able to
respond to strategies 80 Approved .000
PO8،AI4،DS1،
DS2،DS7،DS8
،DS10،DS13
Ensure end users satisfaction from presented
level of services 100 Approved .000
PO9،Ds10،
ME2
Protect achieved objectives 70 Isn’t Approved .045
PO8،PO10 Delivery of services in required time and
budget and quality standards 100 Approved .000
PO9،DS5،DS9
،DS12،ME2
Protect IT assets 80 Approved .005
PO8،AI4،AI6،
AI7،DS10
Decrease defects of service delivery and
solutions 90 Approved .002
PO6،AI6،DS4،
DS12
Ensure minimum effect of defects and
changes of IT services in business 80 Approved .012
AI5 Create and maintain standard and integrated
IT infrastructure 70 Approved .002
AI6،DS5 Maintain information integrity and
processing infrastructure 80 Approved .018
PO6،AI7،DS4،
DS5،DS12،
Ensure resistance of services and IT
infrastructure from failure, delivered attack 60 Isn’t Approved .621
- 15 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
DS13،ME2 and disaster
DS3،DS4،DS8
،DS13
Ensure service availability 100 Approved .000
DS11،ME2،
ME3،ME4
Ensure IT compliance with rules and
contracts 50 Isn’t Approved .725
As it’s mentioned above, 7 first priorities between approved objectives were selected as key objective
in the aspect. Then by using Friedman test the objectives were prioritized. Results were shown for
“customers and users” aspect in Table 2. Hypothesis for Friedman test are as follows:
H0: there is no significant difference between objectives
H1: there is significant difference between objectives
Table 2. Friedman test for prioritizing objectives of “customers and users” aspect
Objective Score Rank
1 Clarify expenses, benefits, strategies,
policies and level of IT services 13.5 6
2 Optimize using of information 15 4
3 Ensure confidentiality of critical
information 20.7 2
4 Delivery of services in required time and
budget and quality standards 14 5
5 Decrease defects of service delivery and
solutions 19.4 3
6 Ensure service availability 8.5 7
7 Ensure end users satisfaction from
presented level of services 26.9 1
Based on results of Friedman, related objectives of each aspect are determined. Further by using
metrics of first level objectives in COBIT, appropriate measures of objectives in each aspect are
determined. Results are shown in table 3.
- 16 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
Table 3. Combined framework of IT-BSC and COBIT
Customers and users Company contribution
measures objectives measures objectives
C21- percent of user’s
satisfaction from
information model and
data availability
C22- percent of
satisfied IT personnel
C23- percent of
satisfied stakeholder
from quality of IT
services, suppliers and
data integration in new
systems
C24- percent of users
that are satisfied by
achieved performance
C25- percent of
satisfied stakeholder
from service delivery in
agreed level
C26- percent of
satisfied users from
service delivery in
agreed level
C27- number of
discontents from service
contracts
C28- percent of user
satisfaction increment
from services, systems
and new technologies
C29- degree of user’s
improvement in result
of better understanding
of systems
1- Ensure service
availability
2- Clarify expenses,
benefits, strategies,
policies and level of IT
services
3- Delivery of services
in required time and
budget and quality
standards
4- Optimize using of
information
5- Decrease defects of
service delivery and
solutions
6- Ensure
confidentiality of
critical information
7- Ensure end users
satisfaction from
presented level of
services
C11- degree of
approval of business
owners from IT
strategic and
operational programs
C12- degree of
alignment with
governance and
business requirements
C13- satisfaction level
of business from
current state of projects
and programs
C14- percent of IT
investment or
provision of
predetermined business
benefits
C15- percent of IT
value drivers mapped
to business value
drivers
C16- percent of IT
expenses that are stated
by value drivers of
business
C17- percent of IT
projects that provide
business expectation in
determined timeline
1- Respond to
business
requirements align
with business
strategy
2- ensure integrating
of application
programs with
business processes
3- ensure automating
of business
transactions and
reliability of
information transfer
4- Ensure cause and
effect relation of
control and
functional
requirements with
effective automated
solutions
5- Clarify the effects
business risks in IT
objectives and
resources
6- Clarify expenses,
benefits, strategies,
policies and level of
IT services
7- IT share in
business profitability
Future orientation Operational excellence
measures objectives measures objectives
C41- percent of
satisfied business
owners from training of
application programs
and supporting issues
C42- number and type
of deviations from
infrastructure plan
C43- level of
stakeholder satisfaction
from personnel’s IT
skills
C44- number of
supported business
processes by old
1- Ensure continuous
improvement and
readiness for change in
future
2- Optimize IT
infrastructures,
resources and
capabilities
3- Ensure appropriate
use and performance of
technology and
application’s solutions
4- Create and maintain
IT skills that are able to
respond to strategies
C31- percent of IT
investment or
provision of
predetermined business
benefits
C32- percent of IT
value drivers mapped
to business value
drivers
C33- percent of IT
expenses that are stated
by value drivers of
business
C34- amount of IT
personnel transactions
1- Respond to
employers
governance
requirements
2- Protect IT assets
3- Protect achieved
objectives
4- Ensure
confidentiality of
critical information
5- Ensure resistance
of services and IT
infrastructure from
failure, delivered
attack and disaster
- 17 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
infrastructure
C45- number of
systems that cover
security requirements
C46- percent of IT
compliance with rules,
regulations and
contracts
C47- degree of senior
satisfaction from
internal and external
monitoring reports
C48- percent of spend
expenses from
competitive
procurement
5- Create and maintain
standard and integrated
IT infrastructure
6- Create and maintain
standard and integrated
application systems
7- Ensure IT
compliance with rules
and contracts
C35- percent of critical
IT objectives that are
covered by risk
evaluation
C36- percent of IT risk
evaluation integrated
with IT risk evaluation
approach
C37- number of
instances were
confidential
information was
formed
C38- number of
business errors in result
of IT services errors
C39- level of
understanding from
expenses, benefits,
strategies, policies and
IT service level
C310- number of
changes in objectives
due to efficiency and
effectiveness of IT
processes
6- Create agility in IT
processes
7- Ensure minimum
effect of defects and
changes of IT
services in business
6. Evaluation of a smaple project using the proposed framework
In this section "system of electronic correspondences" in a research organization will be evaluated by
using the measures obtained in the proposed framework. This project is designed and implemented to
make the process of sending and receiving e-mails within the organization electronic and accelerates
process of responses to the e-mails. This project has established about three years and makes the
electronic communications of 450 staff and managers in Iran Telecommunication Research Center. To
evaluate the measures, a questionnaire to suit each of the measures in the various aspects was designed,
so that the degree of realization of any measure in qualitative form in mentioned project was called into
question. The questionnaires included 34 questions and were distributed between two groups of
respondents. The first group consisted of five experts including: project supervisor, two consulter,
managing director and project manager to answer three aspects of “company contribution”, “operational
excellence” and “future orientation” and the second group consisted of 30 users (Personnel) to respond
to questions in “users and customers” aspect. Responses about the agreement or disagreement were
analyzed using a five point Likert scale. The extent to which the respondents agrees or disagrees on the
realization or non-realization of the measures in the project is graded on a scale of 1 to 9, where
1=strongly isn’t fulfilled, 3=isn’t fulfilled, 5= almost fulfilled, 7= fulfilled, 9= strongly fulfilled. An
average, standard deviation, mod and normal weight are also calculated for each aspect. The higher the
average score, the most likely the measure is close to realization [46][47]. Further, the reliability or
internal consistency of questionnaire was assessed by Cronbach’s alpha [49]. Results are shown in
Table4.
Analyses were done as follows: The mean scores from 1 to 4 means "weakness in the realization of
system measures". Mean score from 4 to 6 (except 6) is assessed as "average". The mean score equal to
6 and above means "strong" and it shows that the project fulfills the criteria to a high level. The results
of averages and weights are shown with two radar diagram in figure 2. Figure 2 shows the degree of
realization of each measure.
- 18 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
Table 4. Results achieved from the project evaluation in organization
aspect measure Average Standard
deviation mode Normal
weight
Overall
weight
score
Company
contribution
C11 8.2 .98 9 .16
6.59
C12 7.4 .80 9 .18
C13 6.2 .98 7 .13
C14 5.8 .98 7 .13
C15 5.8 .98 7 .14
C16 5.4 .80 7 .11
C17 6.6 .80 7 .15
Users and
customers
C21 4.6 1.50 7 .10
5.68
C22 5.4 .80 7 .12
C23 5.8 .98 7 .12
C24 5.8 2.04 9 .12
C25 5.4 .80 7 .10
C26 6.2 1.60 9 .11
C27 5 1.26 7 .10
C28 7 1.26 9 .10
C29 5.8 .98 7 .12
Operational
excellence
C31 5.8 .98 7 .10
5.53
C32 5.8 .98 7 .09
C33 5.4 .80 7 .10
C34 7.4 .80 9 .09
C35 5 1.26 7 .09
C36 4.6 1.50 7 .10
C37 3.4 .80 5 .07
C38 6.6 .80 7 .11
C39 6.2 .98 7 .13
C310 4.6 1.50 7 .12
Future
orientation
C41 6.2 .98 7 .15
5.07
C42 3.8 .98 5 .11
C43 6.2 1.60 9 .11
C44 2.2 1.60 5 .09
C45 4.2 .98 5 .14
C46 5.8 .98 7 .13
C47 6.6 1.50 9 .15
C48 4.2 1.60 7 .12
As it’s shown in the table the “company contribution’ aspects has higher score. This shows that the
mentioned project realize this aspect more than other ones. Realization of “users and customers”,
“operational excellence” and “future orientation” aspects are in the next order.
- 19 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
Figure 2. Realization degree of evaluating measures in project (scores below 4 is weak, scores between
4 and 6 is medium, and scores above 6 is strong)
7. Conclusion
Today information is the most important asset and strategic resources in any organization. Therefore,
management of IT is the major basis of organizational planning and the need to ensure the value of IT,
related risk management and information control are considered as key element in the governance of any
organization. IT project have specific characteristics such as high risk, limited return on investment,
having intangible results, high costs and hidden benefits that make the use of traditional methods to
measure the benefits of the project very difficult. This article provided a framework for evaluating IT
project with combining IT-BSC and COBIT framework. Aspects of proposed framework are chosen
from IT-BSC and objectives and related measures extracted from COBIT and validated using binomial
test and survey of experts in the field of information and communication technology. Since strategic
alignment issue is one of the most important concerns of today’s IT managers, this framework is greatly
contributed to alignment of IT and business in the organization. These measures can be used to evaluate
all IT projects and assess the value of various projects to each other in an organization. For this purpose
it’s necessary to provide weights to four aspects of "company contribution", "users and customers",
"operational excellence" and "Future orientation" in accordance with vision and strategies of
organization and then the mean weight scores according to weight of aspects are determined. So it will
give only one score to each project that makes it comparable with other projects.
8. References
[1] Hochstrasser B., Griths C., “Controlling IT investment: strategy and management”, Chapman &
Hall, London, 1991.
[2] Bacon J., “Why companies invest in information technology”, In: Willcocks L, editor. Information
management: the evaluation of information systems investments. Chapman & Hall, London, 1994.
[3] Willcocks L., Lester S., “The evaluation and management of information systems investments:
from feasibility to routine operations”. In: Willcocks L, editor. Investing in information systems:
evaluation and management. Chapman & Hall, London,1996.
[4] Fitzgerald G., “Evaluating information systems projects: a multidimensional approach”, Journal of
Information Technology, Vol.14, pp.17–30, 1998.
[5] Willcocks L., Lester S., “Evaluating the feasibility of information systems investments: recent UK
evidence and new approaches”, In: Willcocks L, editor. Information management: the evaluation
of information systems investments. Chapman & Hall, London, 1994.
[6] Earl JM., “Management strategies for information technology”, Prentice Hall, New York, 1989.
[7] Strassmann PA., “The squandered computer: evaluating the business alignment of information
technologies”, The Information Economics Press, New Canaan, CT, USA, 1997.
[8] Hinton CM, Kaye GR., “The hidden Investments in Information Technology: the role of
organizational context and system dependency”, International Journal of Information
Management, Vol.18, No.6, pp.413–27, 1996.
- 20 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
[9] Ballantine J., Stray S., “Financial appraisal and the ICT investment decision making process”,
Journal of Information Technology, Vol.13, pp.3–15, 1998.
[10] Apostolopoulos TK, Pramataris KC., “Information technology investment evaluation:
telecommunication infrastructure”, International Journal of Information Management, Vol.19,
No.4, pp.287– 96, 1997.
[11] Brealey RA, Stewart CM, Marcus AJ, “Fundamentals of corporate finance”, McGraw-Hill, New
York, 1995.
[12] Manigart S, Ooghe H., “Investment selection methods”, Management for engineers, vol.11, No.4,
pp.1–18, 1994.
[13] Lumby s., “Investment appraisal and related decisions”, Van Nostrand Reinhold, London, 1981.
[14] Clemons EK, Weber BW, “Strategic information technology investments: guidelines for decision
making”, Journal of Management Information Systems, Vol. 7, No.2, pp.10–31, 1990.
[15] Turner JR, “The handbook of project-based management”. McGraw-Hill, New York, 1993.
[16] Ballantine JA, Galliers RD, Stray SJ, “Information systems/technology evaluation practices:
evidence from UK organizations”, Journal of Information Technology, Vol.12, No.1, pp.29–41,
1996.
[17] Wateridge J., “Training for IS/IT project managers: a way forward”, International Journal of
Project Management, Vol.15, No.5, pp.283-288, 1997.
[18] Wright JN, “Time and budget: the twin imperatives of a project sponsor”, International Journal of
Project Management, Vol.15, No.3, pp.181–186, 1997.
[19] Willcocks L., “Information management: the evaluation of information systems investments”,
Chapman & Hall, London, 1994.
[20] Dos Santos B., “Justifying investments in new information technologies”, Journal of Management
Information Systems, Vol.7, No.4, pp. 71–90, 1991.
[21] Whiting R., Davies J., Knul M., “Investment appraisal for IT systems”, In: Willcocks L, editor.
Investing in information systems: evaluation and management, Chapman & Hall, London, 1996.
[22] Strassmann P., “The business value of computers”, The Information Economics Press, New
Canaan, CT, USA, 1990.
[23] Porter M., “Technology and competitive advantage”, Journal of Business Strategy, Vol. 5, No. 3,
pp.60-78, 1985.
[24] Shank JK, Govindarajan X., “Strategic cost analysis of technological investments”, Sloan
Management Review, Vol.34, No.1, pp.39–51, 1992.
[25] Parker MM, Trainor HE, Benson RJ, “Information strategy and economics: linking information
systems strategy to business performance”, Prentice Hall International, 1989.
[26] Wiseman D., “Information economics: a practical approach to valuing information systems”, In:
Willcocks L, editor. Information management: the evaluation of information systems investments.
Chapman & Hall, London, 1994.
[27] Moad J., “Time for a fresh approach to ROI”, Datamation archive, Vol.41, No.
3, pp. 57-59, 1995.
[28] Willcocks L., “Investing in information systems: evaluation and management”, Chapman & Hall,
London, 1996.
[29] Kaplan RS, Norton DP. “The balanced scorecard: measures that drive performance”, Harvard
Business Review, pp.71–80, 1992.
[30] Stewart W.E., “Balanced scorecard for projects”, Project Management Journal, Vol.32, No.1,
pp.38–53, 2001.
[31] Barber, E., Miley, F., “Monitoring Project Progress: More than a series of feedback loops”,
Australasian Evaluation Society International Conference, October/November 2002.
[32] Stewart W.E., “Balanced scorecard for projects”, Project Management Journal, Vol.32, No.1,
pp.38–53, 2001.
[33] Eilat H., “R&D project evaluation: An integrated DEA and balanced scorecard approach”, Omega,
Vol.36, No.5, pp.895-912, 2006, DOI: 10.1016/j.omega.2006.05.002.
[34] Van Grembergen, W., Saull, R., “Information Technology Governance through the Balanced
Scorecard”, Proceedings of the 34th Hawaii International Conference on System Sciences
(HICSS), pp 1-12, 2001.
- 21 -

Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT
Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi
International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011
[35] Milis. K, Mercken, R, “The use of the balanced scorecard for evaluation of information and
communication technology projects”, International Journal of Project Management, No. 22, pp.
87-97, 2004.
[36] Dickson G. Wetherbe J., “The Management of Information Systems”, McGraw Hill, New York,
1985.
[37] Grembergan. “The Balanced Scorecard and IT Governance”, Information Systems Control Journal,
Vol.2, pp.40-43 ,2000.
[38] Son, Sertac,Weitzel. “Designing a Process Oriented Framework for IT Performance Management
Systems”, The Electronic Journal Information Systems Evaluation, Volume 8, No. 3, pp. 219-228,
2005.
[39] Sedera; Gable;Rosemann;. “A Balanced Scorecard Approach to Enterprise Systems Performance
Measurement”, Proceedings of the Twelfth Australasian Conference on Information Systems.
2001.
[40] Van Grembergen, W., De Haes, S., “Measuring and Improving IT Governance through the
Balanced Scorecard." Information Systems Control Journal, Vol. 2, 2005.
[41] Grembergen. “IT Governance and its Mechanisms”, 35th Hawaii International Conference on
System Sciences (HICCS). IEEE, 2002.
[42] Salle, Mathias, “IT Service Management and IT Governance: Review, Comparative Analysis and
their Impact on Utility Computing’, HP Labs Technical Report HPL-2004-98.
[43] Zee, Van der. “Alignment is not enough: integrating business and IT Management with the
Balanced Scorecard”, Proceedings of the 1st Conference on the IT Balanced Scorecard. antwerp,
1999.
[44] COBIT 4.1, IT governance institute, USA, 2007.
[45] Keeney, R. L., “Value-Focused Thinking: A Path to Creative Decision Making” .Cambridge, MA:
Harvard University Press, 1992.
[46] Ruikar, K., Anumba, C., Carrillo, P., “VERDICT-an e-readiness assessment application for
construction companies”, Automation in construction, Vol. 15, No.1,pp 98-110, 2006.
[47] Mohammad Kazem HAKI, Maia Wentland Forte, “Service-Oriented Business-IT Alignment: a
SOA Governance Model”, AISS: Advances in Information Science and Service Sciences, Vol. 2,
No. 2, pp. 51 - 60, 2010.
[48] Johannes K. Chiang, "E-Commerce Research Trend Forecasting: A Study of Bibliometric
Methodology", JDCTA: International Journal of Digital Content Technology and its Applications,
Vol. 5, No. 1, pp. 101 - 111, 2011.
[49] Zhou Ping, Le Zhong-Jian, "Analyzing Allocation Strategies of Government Information
Resources Using System Dynamic Feedback Archetypes", JDCTA: International Journal of
Digital Content Technology and its Applications, Vol. 5, No. 1, pp. 296 - 304, 2011.
- 22 -