Available via license: CC BY 4.0
Content may be subject to copyright.
Information Technology and Management Science
65
ISSN 2255-9094 (online)
ISSN 2255-9086 (print)
December 2016, vol. 19, pp. 65–70
doi: 10.1515/itms-2016-0013
https://www.degruyter.com/view/j/itms
©2016 Māris Riņģis, Solvita Bērziša.
This is an open access article licensed under the Creative Commons Attribution License
(http://creativecommons.org/licenses/by/4.0), in the manner agreed with De Gruyter Open.
Efficiency Measurement of Project Management
Software Usage at State Social Insurance Agency
Māris Riņģis1, Solvita Bērziša2
1, 2 Riga Technical University
Abstract – One of the activities for improvement of project
management (PM) quality is to introduce or change PM software
to a more suitable one for an appropriate project and team. After
implementation of the new PM software, it is useful to
understand real improvement and effectiveness of the PM
software usage. The objective of the present research is to
identify characteristics and methods for effectiveness
measurement of the PM software usage and demonstrate its
application for the case study to measure project team efficiency
after the PM software implementation and assess impact of the
PM software usage on the project performance. Mann-Whitney
test and Spearman correlation coefficient have been used to
analyse relevance between the PM software usage characteristics
and project performance indicators with the purpose to
determine the PM software usage impact on the project
performance and PM quality. The case study has been performed
at the State Social Insurance Agency. In order to increase PM
quality, Redmine (free of charge PM software) has been
implemented.
Keywords – Efficiency measurement, Mann-Whitney test,
project management software, Redmine, Spearman correlation
coefficient.
I. INTRODUCTION
With an increase of information technology (IT) usage,
inside a company IT implementation in daily processes and
fast and effective decision making become of great importance
[1],[3]. One of the important keys in successful achievement
of this target is well-organised project management (PM) of IT
implementation projects. It could be reached by using project
management information systems (PMISs) during project
planning, execution and control. PMIS affords tools to a
project manager for team managing and provides a company
with new information and knowledge to which managers
should pay special attention [3].
PMIS provides a company with several advantages (quality
improvement, control, scheduling, evaluation, staff
management, and risk management), but not always the usage
of PMIS transforms in positive gain [4], [12], [6]
Unsuccessfully chosen PMIS could not give the desired
improvement in PM, project progress and team performance
that may be related to difficulties of PMIS usage due to a lack
of knowledge or PMIS does not fit the project needs. Not
always companies measure project characteristics and evaluate
real effectiveness improvement by using PMIS after
implementation of PMIS and using it for some time. As
efficiency is an undefined and unknown indicator, it is hard to
determine if PM becomes more qualitative and the project
team – more effective. To make sure about project
performance, it is necessary to find out its metrics, make
measurements and determine impact on project performance.
Thus, the objective of the present research is to identify
characteristics and methods for effectiveness measurement of
the PMIS usage and demonstrate its application for the case
study to measure project team efficiency after the PM software
implementation and assess impact of the PMIS usage on the
project performance.
For the industry case study, the authors have used the data
warehouse team of the State Social Insurance Agency (SSIA).
With the purpose to increase PM efficiency of the SSIA data
warehouse team, Redmine [6] was configured according to
project and team requirements and implemented one year
before evaluation.
The rest of the paper is structured as follows: Section 2
describes the related research on efficiency measurement of
PMIS usage identified using a systematic literature review;
Section 3 presents a case study description and procedure of
measurement collection and analysis; results of the case study
are given and discussed in Section 4. Conclusion and
limitation of the research are discussed at the end of the paper.
II. RELATED RESEARCH
To identify characteristics and methods that can be used for
the effectiveness measurement of the PMIS usage, we have
investigated the existing studies in this direction. To make a
literature analysis enough effective, the systematic literature
review [8] principles have been used. The process of literature
review has been described in Section II.A, and the results have
been discussed in Section II.B.
A. Process
Following the systematic literature review [8] principles,
the following steps have been performed.
Step 1. Research questions have been determined to which
answers we would like to find:
How could performance of PMIS be valued and what
metrics have been measured and analysed?
What is the PMIS impact on a project?
What information could a company obtain after PMIS
implementation?
What is the experience of PMIS implementation in data
warehouse?
What are the prerequisites for correct measurement of
PMIS performance?
Step 2. Keywords for related article search have been
defined:
"Project management tools efficiency metrics"
OR "Project management tools efficiency factor" OR
"Measuring project management tools efficiency" OR
Unauthenticated
Download Date | 1/20/17 10:19 PM
Information Technology and Management Science
_______________________________________________________________________________________________ 2016/19
66
"Increasing efficiency of project management" OR
"Implementing project management tool" OR
"Successful project management tool implementing" OR
"Advantage of project management tools" OR "Project
management tools impact" OR "Project management
information systems efficiency factor" OR "Project
management information systems effectiveness factor"
OR "Measuring project management information systems
efficiency" OR "Implementing project management
information systems" OR "Project management tools"
OR "Project management information systems" OR
"Project management software" OR "Project management
system" OR "Project management software deployment
in data warehouse".
Step 3. Article search in scientific databases have been
performed. The four databases (IEEE Xplore, Science Direct,
Web of Science, Springer, Scopus) have been used, and
articles containing the mentioned keywords in headings,
abstracts or metadata have been selected. Results of article
search are summarised in Table I.
TABLE I
RESULTS OF RESEARCH ARTICLES
NO. DATABASE FORMAT ARTICLE COUNT
1. ScienceDirect Text 1053
2. Scopus Text 3887
3. Web of Science Text 895
4. IEEE Xplore Text 1854
Articles have been classified by year to find out the
popularity of research topic. Article distribution by the period
of publication in Scopus is shown in Fig. 1. It shows that the
topic of PMIS implementation and efficiency was popular
after 2007, when the popularity of the PM software increased
[10].
Fig. 1. Research article count by year in Scopus database.
Distribution of articles by science fields in Scopus is
illustrated in Fig. 2. Articles with the above-mentioned
keywords have frequently been met in computer science,
engineering science and business.
Step 4. Article filtering according to inclusion requirements:
Article contains information about at least one PM
efficiency measurement method, process and results;
Described tools and methods must be related to
information technology management;
All content must be available in full text.
Fig. 2. Division of research articles by science fields in Scopus
(1995–2016).
After article filtering, about 1 % of articles have remained.
The number of articles has noticeably decreased after applying
the third inclusion requirement to 100 articles.
B. Results
Researches that have been related to IT/IS impact on
company performed measurement and analysis to determine
influence on productivity, process optimization, innovation
implementation. Some studies describe only IT influence on a
company, for example, supply channel or country’s economic
development [10], [11]. In [12], the relation between users’
satisfaction with PMIS and PMIS itself and its quality is
described, and it has been detected that PMIS quality
significantly influences users’ satisfaction that is the primary
issue of PMIS effective use. In one of the studies, it has been
determined that PMIS provides the project manager with such
advantages as project planning, monitoring, controlling, and
decision making [13]. Positive impact on PMIS performance
is exerted by users who are well informed about data managed
in PMIS [14].
Methods like Mann-Whitney U-test and Spearman
correlation coefficient have been used for the purpose of
analysis [11]. Mann-Whitney metric is a non-parametric test
of differences in means, and it is used since the distribution of
the population is unknown and the sizes of three subsamples
are small. Spearman correlation coefficient is a non-
parametric measure of rank correlation. This metric is used to
evaluate dependence of two indicators if normal distribution
has not been proven and one or both indicators have been
evaluated using range scale [15]. To make calculations, the
working table is created where the first and the second
columns consist of related independent and fruitful indicator
values – xi and yi. The third and the fourth columns contain
values of ranges – gx.i and gy.i. The fifth column contains
related match range differences – di = gx.i - gy.i. The sixth
Unauthenticated
Download Date | 1/20/17 10:19 PM
Information Technology and Management Science
_______________________________________________________________________________________________ 2016/19
67
column contains square di2. Spearman correlation coefficient
is calculated as follows:
2
2
6
1(1)
i
s
d
rnn
. (1)
Among the found articles, there is no article that describes
PMIS implementation impact on data warehouse PM.
Moreover, the found articles are not mostly based on real data.
Data warehouse project is specific as one kind of its
developing products is universes and reports based on them.
III. CASE STUDY DESIGN
The case study has been performed in SSIA data warehouse
projects where free of charge PM software Redmine was
implemented one year before evaluation (detailed description
of case has been given in Section 3.A). During the evaluation
procedure, data collection and analysis activities have been
performed (Section 3.B) to measure project team performance
changes and assess impact of the PM software usage on
project performance after Redmine implementation.
A. Case Description
Redmine was implemented to prevent SSIA data warehouse
team from the following problems:
Project manager did not know what exact activities were
made by team members in each case;
There was no common overview of the completed and
scheduled tasks;
It was impossible to manage the team effectively enough
because of lack of time management tools, etc.
Redmine is used for project, incident and risk management.
This tool is open source and widely customisable. It was
written by Jean-Philippe Lang in 2006 [6]. Redmine is a web
based application whose server could be placed both on
Windows and Linux, Unix, CentOS, etc. Redmine stores its
data in one of the following databases – MySQL, PostgreSQL
or SQLite. It uses Ruby on Rails framework and one of the
following web servers – Apache, Apache, Ngnix, and
Webrick.
As shown in Fig. 3, Redmine consists of four modules
where each is responsible for specific information part. Project
documentation module includes document adding, deletion
and editing. This module also includes Wiki pages where
tutorials and tips are stored. Management information module
includes task, its progress management and time logging. PM
module includes internal repository and event management.
Communication module is responsible for any kind of contacts
between team members. This module is well developed
providing the members involved with email messaging for
task status or field value change.
Fig. 3. Modules of Redmine.
B. Data Collection and Analysis Procedure
The study consists of two flows shown in Fig. 4 where the
first provides steps for data collection and the second –
analysis.
The data collection flow describes all data preparation steps
and compilation formulas (Fig. 4). Three global variables have
been chosen for data collection that consists of few metrics.
Global variables describe PM and project indicators, which are
shown in Table II. The metrics used in this case study has
been chosen in accordance with similar studies about the PM
software / PMIS impact on project [16], [6].
Fig. 4. Data collection and analysis procedure.
Unauthenticated
Download Date | 1/20/17 10:19 PM
Information Technology and Management Science
_______________________________________________________________________________________________ 2016/19
68
The data collection includes the following steps (Fig. 4).
Step 1: Project data collection and compilation. The number
of involved team members and duration of project have been
used for evaluation of the project metrics. Product is a unit
processed inside a project and described by product
complexity that is a sum of product objects, formulas, reports
divided by duration of a project.
TABLE II
CASE STUDY METRICS
PROJECT METRICS PM SOFTWARE
METRICS
PROJECT
PERFORMANCE
Involved member count
(full time) Frequency of use Cost performance
index
Duration of project
(days) Duration of use (days)
Product complexity
Step 2: Project performance data collection and
compilation. Cost performance index (CPI) has been used as
the project performance metrics that corresponds to the ratio
of the budget cost of work performed to the actual cost of
work performed.
Step 3: Project classification. In the data warehouse project,
threshold values of CPI (a = 0.9 and b = 0.8) have been
defined by the project manager that divides projects into
groups by CPI according to Table III.
TABLE III
PERFORMANCE LEVELS
COST
PERFORMANCE
INDEX
DESCRIPTION PROJECT GROUP
NAME
CPI = 1 According to budget
CPI < b Corrective measures needed weak CPI, WCPI
b <= CPI < a Improvements required study CPI, SCPI
CPI >= a Good budget usage good CPI, GCPI
Step 4: Statistical data collection and compilation of the PM
software by modules. PC Pandora was used for frequency and
duration of use measurement of each Redmine module, which
could log all computer usage statistics and provide HTML
type reports. Program was configured to log hits on websites
and spent time inside each of it. As each module has a definite
site so that it was easy to determine and measure the intensity
of use and spent time.
The PM software metrics have been evaluated by division
of frequency of use of modules and duration of a project in
days as well as by division of duration of use of module and
duration of a project.
The second flow (Fig. 4) supports the analysis of collected
data. The analysis is based on methodology, which assumes
positive relation between an ideal profile and performance
[17]. In order to develop an ideal profile, it is necessary to use
system metric mean score values and divide them into
appropriate performance group. Such an empirically derived
profile is close to the concept of strategic benchmarking,
rather straightforward and intuitively appealing [17]. To detect
an ideal profile, mean duration of tool usage has been
determined.
The analysis includes the following four steps (Fig. 4).
Step 1: Analysis of usage frequency of the PM software
modules comparing metrics by performance groups (weak
CPI, study CPI, good CPI) with Mann-Whitney test. This test
has been chosen because distribution of the population is
unknown and the sizes of the three groups are small. Mean
usage frequency of the PM software modules for each
performance group has been determined by dividing
connection count of module in each project by project duration.
The performance groups have been compared using Mann-
Whitney test for each module to determine difference
significance of value distribution of mean usage frequency.
Step 2: Analysis of usage duration of the PM software
modules comparing metrics by performance groups (weak
CPI, study CPI, good CPI) with Mann-Whitney test. Mean
usage duration of the PM software for each performance group
has been determined by dividing duration of connection for
each project by its duration. Then values of performance
groups have been compared using Mann-Whitney test to
determine difference significance of value distribution of
mean usage duration.
Step 3: Analysis of mean scores of usage frequency of PM
software and product complexity correlation by performance
groups (weak CPI, study CPI, good CPI) using Spearman test.
First, mean values of previously determined mean usage
frequency of the PM software modules have been calculated.
Then these mean value have been compared with product
complexity by the performance groups using Spearman test.
Step 4: Analysis of usage frequency of the PM software
modules and CPI correlation by performance groups (weak
CPI, study CPI, good CPI) using Spearman test. The
previously determined mean values of usage frequency of the
PM software module have been compared with CPI values
using Spearman test. The positive correlation values indicate
that the more tools are used the better project performance is.
IV. CASE STUDY RESULTS
This section describes results of the case data collection and
analysis according to the previously defined procedure
(Section II.B).
The case study metrics have been collected from
15 different projects. Each project used the same Redmine
modules: Project documentation, Management information,
Project management, Communication. Statistics were analysed
by MS Excel. Table IV demonstrates mean score and standard
deviation of main project metrics.
Table V shows mean scores of usage frequency of Redmine
modules by project groups (WCPI, SCPI, GCPI) as well as
significance levels (p) according to Mann-Whitney test (non-
parametric test of differences in means). Grouping of projects
has been made according to CPI threshold values: WCPI
include 4 projects, SCPI – 7 projects, GCPI – 4 projects.
Unauthenticated
Download Date | 1/20/17 10:19 PM
Information Technology and Management Science
_______________________________________________________________________________________________ 2016/19
69
TABLE IV
PROJECT METRICS
P
RODUCT
COMPLEXITY
INDEX
DURATION
OF PROJECT
INVOLVED
MEMBER COUNT
(FULL TIME)
CPI
Mean score 6.13 50.73 days 5.93 count 0.92
Standard
deviation
2.36 21.77 2.28 0.21
Observation
count
15 15 15 15
TABLE V
MEAN SCORES OF USAGE FREQUENCY OF REDMINE MODULES
REDMINE
MODULES
MEAN SCORES MANN-WHITNEY
WCPI (1) SCPI (2) GCPI (3) 1–2 1–3 2–3
Project
documentation 14.274 25.385 17.891 0.007
*** NS NS
Management
information 19.371 29.513 24.474 0.007
***
0.010
** NS
Project
management 23.292 32.014 26.657 0.019** NS NS
Communication 8.444 15.431 14.224 0.019** 0.005
*** NS
NS Not significant, *p < 0.10, **p < 0.05, ***p < 0.01
Results show that comparing WCPI and GCPI (ideal profile
scores) for 2 of 4 modules a significant difference has been
observed. Comparing WCPI and SCPI, significant changes
have been observed for all modules. Mean metric scores of
SCPI group are greater than ideal profile (GCPI), which
means that the stated level of Redmine use does not allow
reaching ideal profile results due to module usage specifics.
Insignificant difference has also been observed between SCPI
and GCPI.
Mean scores of usage duration of Redmine module in each
project group (WCPI, SCPI, GCPI), as well as significance
level values (p) that have been calculated comparing groups
using Mann-Whitney test are shown in Table VI.
TABLE VI
MEAN SCORES OF USAGE DURATION OF REDMINE
MEAN SCORES MANN-WHITNEY
WCPI(1) SCPI(2) GCPI(3) 1–2 1–3 2–3
0.669 1.050 0.876 0.012** 0.010** NS
NS Not significant, *p < 0.10, **p < 0.05
Table VI shows that WCPI mean usage duration of
Redmine with significance 0.05 differs from other group
results. Insignificant difference has been observed between
SCPI and GCPI despite the fact that SCPI mean score is
greater than others.
Correlation between product complexity index and usage
frequency of Redmine by project groups is demonstrated in
Table VII. We have observed that by increasing project
complexity the usage frequency of Redmine also increases.
The greatest coefficient to GCPI proves this statement.
TABLE VII
CORRELATION BETWEEN PRODUCT COMPLEXITY INDEX
AND USAGE FREQUENCY OF REDMINE BY GROUPS
GROUP CORRELATION
WCPI(1) −0.059
SCPI(2) 0.450
GCPI(3) 0.509
Correlation between usage frequency of Redmine module
and project performance is shown in Table VIII. Correlation
coefficients demonstrate that often the usage of Redmine has
been observed for project with the greatest performance. The
greatest correlation has been observed for a communication
module that also has the greatest impact on project
performance.
TABLE VIII
CORRELATION BETWEEN USAGE FREQUENCY OF MODULES
AND PROJECT PERFORMANCE
REDMINE MODULES CORRELATION
Project documentation 0.053
Management information 0.027
Project management 0.009
Communication 0.202
V. CONCLUSION
This paper demonstrates the case study of impact analysis
of Redmine usage to project performance by using Mann-
Whitney test and Spearman correlation coefficient. Fifteen
projects of data warehouse team have been observed, data
about Redmine use (usage frequency and usage duration) have
been collected, project metrics and product complexity
analysed. The main conclusions of Redmine usage efficiency
evaluation are as follows:
The longer Redmine is used, the better CPI results are,
i.e., the studied tool has a positive effect on the project
success.
Greater correlation was between usage frequency of
communication module and project performance, which
means that this module has the greatest impact on project
performance.
Increasing project performance also increases usage
frequency of Redmine.
Similar results have also been observed in other studies
[18], [4].
One of the limitations of this case study is that duration of
use of a website is not exact enough because a user can open a
website and leave it for a long time and it does not mean that
the page has been really used. Case study has only 15 projects
and it is not enough for general conclusions about PMIS usage
effectiveness improvement. The evaluated projects can be
used as a basis for other studies.
Future studies will be related to adding additional metrics,
analysing another performance indicator instead of CPI.
Unauthenticated
Download Date | 1/20/17 10:19 PM
Information Technology and Management Science
_______________________________________________________________________________________________ 2016/19
70
REFERENCES
[1] N. I. Jabbouri, R. Siron, I. Zahari and M. Khalid, “Impact of Information
Technology Infrastructure on Innovation Performance: An Empirical
Study on Private Universities In Iraq,” Procedia Economics and
Finance, vol. 39, pp. 861–869, 2016.
https://doi.org/10.1016/s2212-5671(16)30250-7
[2] R. Frese, “Project Success and Failure: What is Success, What is Failure,
and How Can You Improve Your Odds for Success?” St. Louis,
Missouri, Dec. 2003.
[3] E. Brynjolfsson and L. M. Hitt, “Beyond Computation: Information
Technology, Organizational Transformation and Business Performance,”
J. of Economic Perspectives, vol. 14, no. 4, pp. 23–48, fall 2000.
https://doi.org/10.1257/jep.14.4.23
[4] R. Pellerin, N. Perrier and X. Guillot, “Project Management Software
Utilization and Project Performance,” Procedia Technology, vol. 9,
pp. 857–866, 2013. https://doi.org/10.1016/j.protcy.2013.12.095
[5] A. Schwer, D. Dufner and Mourcous G, “Project Management
Informatics,” in Proc. of the 2005 Midwest Section Conf. of the
American Society for Eng. Educ., 2005, pp. 1–5.
[6] L. Raymond and F. Bergeron, “Impact of Project Management
Information Systems on Project Performance,” Handbook on Project
Management and Scheduling Vol. 2 (International Handbooks on
Information Systems). Springer International Publishing, 2014,
pp. 1339–1354. https://doi.org/10.1007/978-3-319-05915-0_30
[7] J.–P. Lang, Redmine, 2006. [Online].
Available: http://www.redmine.org/ Accessed on: March 20, 2015.
[8] B. Kitchenham and S. Charters, “Guidelines for performing Systematic
Literature Reviews in Software Engineering,” Keele University and
Durham University Joint Report 2007.
[9] S. Rivard and R. Dupré, “Information systems project management
in PMJ: A brief history,” Project Manage. J., vol. 40, issue 4, pp. 20–30,
Dec. 2009. https://doi.org/10.1002/pmj.20143
[10] C. Marinagi, P. Trivellas and D. P. Sakas, “The Impact of Information
Technology on the Development of Supply Chain Competitive
Advantage,” Procedia – Social and Behavioral Sciences, vol. 147,
pp. 586–591, Aug. 2014. https://doi.org/10.1016/j.sbspro.2014.07.161
[11] H. Son, N. Hwang, C. Kim and Y. Cho, “Construction professionals’
perceived benefits of PMIS: The effects of PMIS quality and computer
self-efficacy,” KSCE Journal of Civil Eng., vol. 20, issue 2, pp. 564–
570, March 2016. https://doi.org/10.1007/s12205-015-0138-1
[12] D. W. Jorgenson, M. S. Ho and J. D. Samuels, “The impact of
information technology on postwar US economic growth,”
Telecommunications Policy, vol. 40, issue 5, pp. 398–411, May 2016.
https://doi.org/10.1016/j.telpol.2015.03.001
[13] L. Raymond and F. Bergeron, “Project management information
systems: An empirical study of their impact on project managers and
project success,” Int. J. of Project Manage., vol. 26, issue 2, pp. 213–
220, Feb. 2008. https://doi.org/10.1016/j.ijproman.2007.06.002
[14] M. G. Kaiser, F. Ahlemann, “Measuring Project Management
Information Systems Success: Towards a Conceptual Model and Survey
Instrument,” Proceedings of the IEEE International Conference on
Software Engineering and Service Sciences, ICSESS, 2010, pp. 259.
Available: http://aisel.aisnet.org/ecis2010/20
[15] J. Dravnieks, Matemātiskās statistikas metodes sporta zinātnē, Riga,
2014, 65.–67. lpp.
[16] A. S. B. Ali, F. T. Anbari and W. H. Money, “Impact of Organizational
and Project Factors on Acceptance and Usage of Project Management
Software and Perceived Project Success,” Project Manage. J., vol. 39,
issue 2, pp. 5–33, June 2008. https://doi.org/10.1002/pmj.20041
[17] É. Lefebvre and L. A. Lefebvre, “Global strategic benchmarking, critical
capabilities and performance of aerospace subcontractors,
Technovation,” vol. 18, issue 4, pp. 223–234, April 1998.
https://doi.org/10.1016/S0166-4972(98)00013-3
[18] P. Fernandes, A. Sales, A. R. Santos and T. Webber, “Performance
Evaluation of Software Development Teams: a Practical Case Study,”
Electronic Notes in Theoretical Computer Science, vol. 275, pp. 73–92,
Sept. 2011. https://doi.org/10.1016/j.entcs.2011.09.006
Māris Riņģis holds a Bachelor degree and is an Information System
Administrator at DNB bank. He obtained his BSc (2014) and will obtain MSc
(2017) degree in Computer Science and Information Technology from Riga
Technical University. From 2013 to 2015, he has worked as an Information
System Administrator at the State Social Insurance Agency. He holds SAP
Business Objects Information Design Tool and SAP Business Objects
administration certificates. He was awarded in RTU contest “Nāc un studē
RTU”.
E-mail: maris.ringis_1@edu.rtu.lv
Solvita Bērziša holds a Doctoral degree and is a Lecturer and Researcher at
the Institute of Information Technology of Riga Technical University (Latvia).
She obtained her Dr. sc. ing. (2012), BSc (2005) and MSc (2007) degrees in
Computer Science and Information Technology from Riga Technical
University. Her main research fields are IT project management, project
management information systems implementation and application, as well as
project data analytics. She also works as an IT Project Manager at Exigen
Services Latvia. She holds PMP and CBAP certificates and was awarded the
IPMA Outstanding Research Contribution of a Young Researcher 2013. She
is a member of PMI and Latvian National Project Management Association.
E-mail: Solvita.Berzisa@rtu.lv
Unauthenticated
Download Date | 1/20/17 10:19 PM
