Content uploaded by Vanessa Nappi
Author content
All content in this area was uploaded by Vanessa Nappi on Nov 21, 2022
Content may be subject to copyright.
KEY PERFORMANCE INDICATORS AND DIMENSIONS FOR THE
INNOVATION PROCESS
Vanessa Nappiᵃ*, Kevin Kellyᵃ
ᵃMechanical and Manufacturing Engineering Department, Trinity College Dublin,
Ireland
*Corresponding author. E-mail addresses: nappiv@tcd.ie
ABSTRACT
Innovation helps promote significant social challenges and at the same time provides
opportunities for established and newly emerging sectors and job creation. Clear
evidence in the literature indicates that their competitive success is dependent upon an
organization’s management of the innovation process. However, to define suitable
indicators to manage performance is a difficult task especially for small businesses and
start-ups. Since performance frameworks and indicators are derived from favourable
results to specifics types of organizations, are not suitable for small businesses as such
practices are initially intended for large organizations. Drawing from literature, the
measurement of the innovation process tends to be infrequently undertaken in an ad hoc
fashion, unbalanced or under-specified frameworks. Therefore, this paper presents a
comprehensive set of key performance indicators for the innovation process into a
performance framework. This research followed a based on a three-step systematic
literature review, followed by a systematization of the key performance indicators into
dimensions for the innovation process. This approach is a well-established procedure
to identify indicators within the new product development domain. As a result, nine
dimensions for the performance framework were identified: innovation strategy;
knowledge management; organization and culture; portfolio management; project
management; technology management; team management; commercialization and
innovation vanguard adoption referring to openness, sustainability and servitization
orientation. Secondly, each dimension of the framework was populated with
corresponding key performance indicators. A total of 146 key performance indicators
were identified and systematized into a rapid assessment particularly fitted for small
and midsized organizations and start-ups and an in-depth approach suitable for large
organizations as well. This paper produces two important contributions. First, it takes
the challenging step of incorporating broad and diverse studies into a single
performance framework fulfilling a gap in the literature. Second, it provides a
performance framework which provides a useful basis for managers in practice to select
suitable indicators to monitor their innovation process, diagnose limitations and
identify areas for improvement actions.
Keywords: Innovation process, performance measurement; key performance
indicators.
2
INTRODUCTION
Innovation is crucial for productivity, competitiveness and economic growth.
Innovation is essential to address significant social challenges and at the same time
promotes opportunities for established and newly emerging sectors and job creation
(EC, 2016). There is a clear association between investment in innovation, growth and
job creation, since innovation accounted for 62% of economic growth in Europe
between 1995 and 2007, according to the “Science, research and innovation
performance of the EU” report (EC, 2016).
Innovation manifests its impact regarding products as the introduction of a good
and services that are new or significantly improved with respect to its original
characteristics or intended uses. This includes significant improvements in technical
specifications, components and materials, incorporated software, user-friendliness or
other functional characteristics (OECD, 2005). While the understanding of innovation
activities and their economic impact has greatly increased in the last decades, it is still
deficient (Adams, Bessant and Phelps, 2006; Crossan and Apaydin, 2010; Nilsson and
Ritzén, 2014; Nicholas et al., 2015). As the world economy evolves, so does the process
of designing an innovative product (Crawford and Di Benedetto, 2011).
Managing innovation is essential for organisations to survive in a competitive
and dynamic environment. Evidence shows that competitive success is dependent upon
an organisation’s management of the innovation process, a multifaceted number of
events and activities occurring concurrently and in sequence at times. One significant
challenge is to measure the complex processes that influence the organisation's
innovation capability (Cordero, 1990; Cooper and Kleinschmidt, 1995; Adams, Bessant
and Phelps, 2006; Zizlavsky, 2015).
For many organisations, managing the innovation process is a complex issue
(Erkens et al., 2014). It is also essential from the academic perspective (Adams, Bessant
and Phelps, 2006; Zizlavsky, 2015). In this way, responding to the need of both
practitioners and academics to understand the effectiveness of innovation actions,
performance indicators are frequently proposed. The underlying reasoning supporting
the use of performance indicators is to establish an objective and a tracking mechanism.
The indicator will support managers to have enough information to make decisions and
take actions to achieve the goals set concerning innovation (Cooper and Kleinschmidt,
1995; Chiesa, Coughlan and Voss, 1996; Crawford and Di Benedetto, 2011).
The approaches towards the use of indicators within the literature on the
management of innovation are fragmented with most studies focusing on the innovation
inputs and outputs, overlooking the processes in-between. With so many performance
indicators presented in the literature, a problem appears to exist for people interested in
using performance indicators to measure the innovation process in place. That is, even
with so many indicators at hand, to define the dimensions that need to be measured and
which ones are the “key” performance indicators (KPIs) to manage the innovation
process is still challenging successfully (Acosta, Araújo and Trabassos, 2002; Adams,
Bessant and Phelps, 2006; Berg et al., 2008; Zizlavsky, 2015). In addition, these
indicators were designed for large companies and can be especially difficult for small
3
companies and start-ups in implementing them, even if with an innovation process in
place (Kleinknecht, 1987; McAdam, McConvery and Amstrong, 2004; Hudson Smith
and Smith, 2007; Katila, Chen and Piezunka, 2012).
Within this context, the objective of this study is to identify dimensions and
performance indicators for a framework to manage the innovation process, not only for
large but also small-sized companies. Therefore, this paper presents a comprehensive
set of dimensions and process-related performance indicator for managing the
innovation process. The major challenge addressed in this research is related to the lack
of systematized approaches that tackle the innovation management to support
companies in the selection of key performance indicators.
It is worthwhile mentioning that this research is prescriptive in nature, since it
is geared towards gathering relevant performance indicators and setting the theoretical
foundation for building future practical applications. This paper is part of a greater
research effort to develop a performance framework for companies monitor and manage
the innovation process, diagnose limitations and prescribe appropriate actions. The
specific application of the indicators in the practical context of the performance
framework is not within the scope of this paper, and will be explored in future research.
In the following section, the research method employed in this research is
presented. Next section describes the primary results obtained by the systematic
literature review, followed by discussions and conclusions.
RESEARCH METHOD
The research method consisted of a three-step systematic literature review. This
systematic review is a method used to map existing and preceding knowledge and
proposals in a specific research domain. Besides the analysis of previous discovery,
techniques, ideas and ways to explore topics, the systematic review also allows the
evaluation of the relevance of information, its synthesis and summarization. The notion
of systematic review has recently gained significance in the management literature to
identify performance indicators for NPD, for instance: lean performance (Mascarenhas
Hornos da Costa et al., 2014); eco-design implementation (Rodrigues, Pigosso and
McAloone, 2016), and environmental performance (Issa et al., 2015). The research
method followed the procedure proposed by (Brereton et al., 2007) based on three main
steps: 1) plan review; 2) conduct review and 3) document review (see Figure 1).
In the first step (plan review), a literature protocol was prepared based on the
research question and definitions expressed through specific concepts and terms known
as the search strings, inclusion criteria and tools and schedule. The second step (conduct
review) refers to the literature search procedures, i.e., the search within the indexed
electronic databases of primary studies, which were then properly assessed according
to the inclusion criteria. As the studies had been selected, relevant data from the
publications, notably the attributes of the KPIs, were recorded, analysed and classified
and synthesized during the third step (document review).
4
Figure 1. Research method with three steps.
The primary objective of the systematic literature review was to identify the
available process-related KPIs for measuring the innovation process. Accordingly, the
selected search keywords are related to KPIs and measurement and innovation process,
and their synonyms. The terms indicator, measure, and metric are often used
interchangeably (Mascarenhas Hornos da Costa et al., 2014). Thus, in this paper, “key
performance indicator” or “indicator” are the terms used its synonyms. To have a more
comprehensive search, the keywords for indicators and innovation process were
searched on the topic, which covers the paper's title, abstract, and keywords.
The process of creating the strings was iterative, in cycles of development,
testing, and refinement. Each iteration involved the transcription of the keywords of the
articles selected in the preliminary search and the selection of relevant terms for this
research within these keywords. It also included the search for synonyms for relevant
terms; the definition of the constraints, expressions that guarantee the right orientation
of the searches; the preparation and test in the indexed electronic database, and finally
the refinement of relevant terms and restrictions and string.
The final string was composed of the term “key performance indicator” and the
synonyms: “indicator,” “measure,” “index,” “indices,” followed by broader terms as
“performance measurement” and “performance evaluation.” Additionally, the
following synonyms of “innovation process” were selected: “innovation”, “innovation
management”, “innovation planning”, “innovation audit”, “front end of innovation” or
“innovation front end”, “fuzzy front end”, “research and development” associated with
other terms “design process”, “product design”, “product development”, “product
service system development”, and “product lifecycle management”. Although these last
associated terms are not usually linked with the innovation process, the creation of the
string showed that they are crucial to providing a more comprehensive search. For
instance, “product lifecycle management” can be defined as an integrated management
approach of product-related information through the entire lifecycle and are highly
associated to boosting innovation in manufacturing companies (Sudarsan et al., 2005).
In the second step, the search used the indexed electronic database ISI Web of
Science (WoS), due to its availability of advanced web search mechanisms, high
volume of indexed publications and proven relevance in this field of research
5
(Adriaanse and Rensleigh, 2013). The studies retrieved from literature should
empirically demonstrate that the KPIs are significant for managing innovation in the
design process. In this way, a more mature and sound database was required, one like
the WoS. The searches included journals, peer-reviewed conference papers, and books
to capture both mature and more recent research under development in distinct fields.
Further, cross-referenced publications were analysed for a more comprehensive review.
The research fields were limited to cover WoS categories of management, business,
planning development, economics, engineering (all kinds), operations research,
computer science, multidisciplinary sciences, and social sciences mathematical
methods. Also, no restriction was made concerning the publication dates in the WoS
database to gain broader results.
In the course of the second step, three inclusion criteria were applied for the
assessment and selection of the publications. First, the publication must contain
quantifiable factor(s) specified in terms of the necessary organizational capabilities to
manage the innovation process. Second, the publications must present, at least, one KPI
for the innovation process. Third, the KPI must focus on the process rather than on the
product itself, meaning that the indicators should be aligned with following stages of
the innovation process: innovation front end, technology development process,
development/design and product accompanying and retirement. Therefore, indicators
dealing directly and exclusively with technicalities of the product, such as physical
characteristics, materials, were not considered. By using these inclusion criteria, the
review intended to cover the publications with proposals of new KPIs as well as those
publications presenting, reviewing, reporting, analysing KPIs from literature. The
procedure for the inclusion assessment was applying the filters followed by the
application of the three inclusion criteria: 1) read the publication’s title; 2) read the
abstract and keywords; 3) read the introduction and conclusion, and 4) read the full
paper. Additionally, this assessment, which is the same procedure from (Mascarenhas
Hornos da Costa et al., 2014; Issa et al., 2015; Rodrigues, Pigosso and McAloone,
2016), can also lead to the identification of other articles employing the cross-
referencing.
In the third step, the selected studies were analysed with the purpose of identifying
the dimensions empirically revealed to be significant in the innovation process.
Meaning that the quantifiable factor(s) must have been specified in measurement,
determined by empirical methodology (either based on case research presenting the
samples with small, medium and large enterprises (SML) companies or survey/expert
assessment). Moreover, the study must have presented clear linkage between with the
innovation process outcomes (based on success rates in sales/profit, on-time and
schedule performance or market objectives). Dimensions were defined as a
categorisation of innovation measurement areas covering a broad spectrum of the in-
between processes, in which the KPIs were grouped (Adams, Bessant and Phelps, 2006;
Crossan and Apaydin, 2010). The concept of theoretical saturation was adopted to
categorize the innovation dimensions, based on previous well-established SLR research
(Glaser and Strauss, 1967; Adams, Bessant and Phelps, 2006). Meaning that a
6
dimension was considered fully explored when no additional data were being found,
and the researcher was empirically confident that the category of the dimension in
question was saturated. Subsequently, the KPIs were identified and documented in an
electronic spreadsheet to function as a database. Capturing the following attributes:
title; purpose; formula; scales proposed (if applicable); relate to (other indicators
associated); comments, and bibliometric information (authors and year of publication)
(Neely, Gregory and Platts, 1995).
Furthermore, the retrieved indicators were further systematized to offer a way to
measure performance by adding two differentiations also to support overlooked small
organizations. The rapid assessment is also mean for the small companies to provide a
way to measure performance dimensions (Czuchry and Yasin, 2001). Rapid assessment
indicators are the ones used to stimulate a quick-win situation for the companies, as a
first “health check-up” to diagnose the current situation. Meanwhile, an in-depth
approach indicator should be used in a later stage analysis, such as used in (Chiesa,
Coughlan and Voss, 1996). To classify into these two differentiations, the procedure
carried out for the indicator’s assignment was based on the searches for specific
keywords in the performance indicators database. These keywords were pulled out from
the dimensions’ definitions and inserted into the search field of the electronic
spreadsheet. They can be further classified in either leading indicators (trends) or
lagging indicators (outputs) (Kaplan and Norton, 1992). Finally, note that the indicators
within the scope of this research are all potential KPIs, once a company selects them
according to their strategy and drivers, since they are identified as crucial to the success
of the innovation process. That is why performance indicators and KPIs are being used
interchangeably during the length of this paper.
RESULTS
The dataset presented in this paper was constructed using the Web of Science
database. Every available publication containing the combination of indicator and
innovation process in its title, keywords or abstract was identified and downloaded.
This search identified 251 papers, refined by the field areas mentioned in the research
method. 240 publications (96% of total) presented the main text in either English or
Spanish/ Portuguese. By applying the inclusion criteria and following the procedure for
selection, 186 documents (74% of total) had their abstract and keywords analysed.
From this sample, 150 publications (60% of total) were available for download. Further,
introduction and conclusion were examined in 69 (27% of total), and 21 papers (8% of
total) were thoroughly read, and finally selected. Figure 2 illustrates a summary of the
results from papers’ analysis and selection.
It is important to highlight that by analysing the selected papers, with their
citations being cross-checked to ensure that any other publications were also captured.
60 publications were elected during this cross-reference analysis. This magnitude is
understandable since a further study of the gathered KPIs showed that the majority of
the selected documents (86%) presented its focus on reporting, analysing, re-defining
the literature rather than originally proposing new KPIs.
7
Figure 2. Graph of the utilization of the SLR extraction from Web of Science.
The selected publications, in addition to the cross-referenced ones, were published
in a total of 39 distinct journals with the majority of 13 papers published and in the
Journal of Product Innovation Management (JPIM). The earliest article included in the
dataset was published in 1982 and the most recent 2016. Figure 3 presents the
distribution of the selected paper and the cross-references regarding publication years.
It is worth to mention that the reasonable increased from the references of the sample,
five, were published within one especial issue addressing the Best Practices in New
Product Development published in the JPIM. Out of 81 publication sources, 76 of them
were retrieved from academic journals, two conference papers, and three books. This
distribution reflects the maturity of the database Web of Science needed for this
research.
Figure 3. Distribution of selected publication by year.
As this research has the focus of gathering available KPIs from multiple sources
in the literature, it should be highlighted that the distribution among different
publication sources is essential, especially when it comes to the innovation process
which is a multidisciplinary concept. This is a proxy for the leading research fields
which are contributing to the proposal of the categorisation of the dimensions, which is
relevant for future studies that will build on top of systematized KPIs. Figure 4 shows
the distribution of papers by journals with at least two publications. JPIM is a well-
8
established journal of innovation management related issues from the operation
management (OM) field. R&D and Research-Technology management represent the
literature about innovation from a techno-centric focus within the 90’s and early 2000.
Engineering sources such as IEEE and the Journal of Engineering and Technology also
presented a techno-centric approach towards innovation and more recently a process
approach. Economics journal also present innovation-related publications, primarily
related to the value creation chain in product development. The remaining journals are
within the scope of management science as expected within the OM and Operations
Research (OR).
Figure 4. Distribution of selected papers by journal.
Based on the selected studies, mostly from OM and OR, evidence shows that there
is diverse literature and as a result, it is hard to operationalize innovation measurement
(Adams, Bessant and Phelps, 2006). Nevertheless, there are areas of commonality
across the literature, and nine dimensions of the innovation process were identified. The
dimensions presented here were empirically demonstrated in the selected studies to be
significant for the management of the innovation in the product development process.
Within these studies, the most relevant ones for identifying the dimensions of the
innovation process are presented in Table 1. These studies were retrieved from the
systematic literature and highlighted here for their importance within the field based on
citation analysis for older studies (at least ten citations) and significance and scope of
the journal for more recent studies (from 2012 onwards).
This sample of studies shows a range of publication years from 1995 to 2016. To
extract dimensions of the innovation process, the concept of theoretical saturation was
adopted to categorize the innovation dimensions, as mentioned in the research method,
based on previous well-established research using SLR (Glaser and Strauss, 1967;
Adams, Bessant and Phelps, 2006). Meaning that an innovation dimension was
considered fully explored when no additional data were being found, and the researcher
9
was empirically confident that the category of the dimension in question was saturated.
In this way, nine dimensions were identified.
In the first column of Table 1, the dimensions of the innovation process derived
from this synthesis are presented: innovation strategy; knowledge management;
organisation and culture; portfolio management; project management; technology
management; team management; commercialization and innovation vanguard adoption
referring to an orientation towards more openness, sustainability, and servitization.
Table 1. Identified dimensions of the innovation process.
References
Dimensions
(Cooper and
Kleinschmidt,
1995)
(Chiesa,
Coughlan and
Voss, 1996)
(Verhaeghe
and Kfir, 2002)
(Adams,
Bessant and
Phelps, 2006)
(Barczak and
Kahn, 2012)
(Nicholas et al.,
2015)
(Lee and
Markham,
2016)
Innovation
strategy
Knowledge
management
Organisation
and culture
Portfolio
management
Project
management
Technology
management
Team
management
Commercialisation
Innovation
vanguard
adoption
Openness
Sustainability
Servitization
One of the most influential studies in new product development measurement
(Cooper and Kleinschmidt, 1995) empirically demonstrated five main constructs for
new product performance across 135 firms: NPD process, NPD strategy, project
management, culture (climate and development team) and management commitment.
Although this framework many contributions’ to the advance of literature, it presents a
bias prevalent in innovation and NPD studies, which is the techno-centric focus. This
narrower focus hinders the incorporating of innovation in non-technical contexts and,
as (Adams, Bessant and Phelps, 2006) states it overlooks other vital factors such as the
role of knowledge.
Another study that influenced many others and also presents the techno-centric
focus is the technical innovation audit (Chiesa, Coughlan and Voss, 1996) tested in
eight SML firms. It describes a framework with two dimensions: inputs with three
enabling processes: human resources, financial resources, adding the new dimensions
10
use of systems and tools and management leadership and four core processes: concept
generation, product development, process innovation and technology acquisition,
already presented in (Cooper and Kleinschmidt, 1995).
Verhaeghe and Kfir, (2002) extended technical innovation audit (Chiesa,
Coughlan and Voss, 1996) to comprehend not only “hard” and technical based products
also more “soft” innovation, either research or consultancy project. It was applied to
the service context resulting in the following main dimensions: leadership, resourcing
innovation, systems and tools, technology transfer and acquisition, market focus,
innovation performance, networking.
Further research based on a synthesis of the previously mentioned studies, Adams
et al., (2006) produced a theoretical proposal that reflects factors apparently significant
to the innovation process. It consists of seven dimensions: inputs, strategy, organization
and culture, portfolio management, project management, and introducing, explicitly,
knowledge management and commercialization. Moreover, although other studies,
such as (Crossan and Apaydin, 2010), have recognized the importance of this study,
these dimensions still need to be applied in real cases.
Barczak and Kahn, (2012) developed a best practice benchmarking framework for
the NPD, using the findings of a survey conducted by the Product Development
Management Association (PDMA) in 2004. This framework includes: strategy,
portfolio management, process, market research, project climate (including project
team), and metrics and performance evaluation. This proposal is particularly interesting
because it comprehends the notion of a progression path, which an organization evolves
thru distinct levels of sophistication. Additionally, the authors introduced the dimension
of market research.
Nicholas et al., (2015), on the other hand, developed their framework with
dimensions identified within the literature related to the development of radical
innovation in products. Afterward, these dimensions were validated in a sample of 87
organisations. The resulting dimensions validated were: market awareness, idea
management, customer involvement, open environment and internal networking.
Furthermore, these authors introduced the factor of open environment, adding the
concept of open innovation into the dimensions of the innovation process.
Finally, based on another survey by PDMA in 2011, (Lee and Markham, 2016)
identified factors revealed to be significant within in the 87 companies. In addition to
the well-established dimensions from the literature (strategy, portfolio management,
fuzzy front end management, project management, organization and culture and
customer focus), the research added new ones: sustainability orientation on new
products, open innovation, and servitization. Especial attention must be given to these
new ones since evidence of their importance have been emerging from diverse fields of
literature (Baines and W. Lightfoot, 2013; Calik and Bardudeen, 2016; Shaner, Beeler
and Noble, 2016).
After that, as already mentioned in the research method section, the performance
indicators were identified, compiled and systematized. To that end, the KPIs were
collected into a database with an electronic spreadsheet format documenting title;
11
purpose; formula; and scales used (if applicable), relate to (other KPIs associated with
this particular indicator); innovation process alignment (innovation front end,
technology development process, development/design and product accompanying and
retirement); comments and bibliometric information (author and year of publication).
Subsequently, they were also assigned to dimensions of the innovation process, using
specific keywords extracted from the dimensions’ terms and inserting them into the
search field of the electronic spreadsheet, the database of performance indicators.
As an example to illustrate the performance indicator assignment, consider the
dimension “innovation strategy.” This dimension deals with planning a synchronized
focus for the new product related processes efforts. Therefore, to assign performance
indicators for this dimension, the studies that were focused on strategy orientation and
leadership (Cooper and Kleinschmidt, 1995, 2007; Tipping, Zeffren and Fusfeld, 1995;
Chiesa, Coughlan and Voss, 1996) were given priority and firstly inspected. Later on,
all other indicators retrieved from the literature were then assessed for alignment with
the dimensions. One of the assigned KPIs was “goals for NPD/innovation effort clear
to everyone involved” for example.
146 unique key performance indicator that met the inclusion criteria were
identified and catalogued. Then, the indicators were consolidated in a single standard
database, representing an average of 1.8 unique KPIs per publication. Almost 90% of
the database were classified as leading indicators rather than lagging as expected, since
they are dealing with the innovation process improvement. Due to its considerable size,
Table 2 presents one example of KPI for each dimension identified, showing: title;
identification number (ID); corresponding dimension; references; and number of KPIs
for the rapid assessment, the in-depth approach, and total. As mentioned previously, the
reason to differentiate is that the need to stimulate a quick-win situation for the
companies, resulting in 34 KPIs for rapid assessment more suitable for small companies
and 112 KPIs more for in-depth analysis. For the previous illustrative case of the
innovation strategy dimension, a total of 13 indicators were identified, being four of
them used for a rapid assessment and the remaining ones used for the in-depth approach.
The results of the distributions of the indicators through dimensions of the
innovation process is also presented in Table 2. The most considerable number of
indicators (15%) addresses the project management and also the technology
management dimension. Second is the indicators regarding knowledge management
(14%), followed by organisation and culture (12%), then by team management (11%),
innovation vanguard adoption (10%), innovation strategy (9%), commercialization
(8%), and portfolio management (5%). The widely recognized idea that project
management excellence promotes the delivery of innovation in product, system, and
service excellence (Barczak and Kahn, 2012) appears to have impacted the proposition
and dissemination of indicators for the innovation process, since, as mentioned, 33% of
the indicators relate to project management. Furthermore, the technology management
dimension presented 15% of total KPIs identified, and as the previous dimension,
technology management indicators are widely spread across the R&D research
literature since the 90s.
12
Table 2. Examples of KPIs for each dimension
Title
ID
Dimension
Reference
Rapid
assessment
In-depth
approach
Goals for NPD/
innovation effort
clear to
everyone
involved
IS1
Innovation strategy
(IS)
(Cooper and
Kleinschmidt,
1995, 2007;
Tipping, Zeffren
and Fusfeld, 1995)
4
9
Total: 13 KPIs (9%)
Percentage of
ideas generated
according to
formal and
informal
activities
KM1
Knowledge
management (KM)
(Eling, Griffin and
Langerak, 2016;
Gurtner and
Reinhardt, 2016;
Lee and Markham,
2016)
3
18
Total: 21 KPIs (14%)
Team Climate
Inventory
OC1
Organisation and
culture (OC)
(Anderson and
West, 1996)
2
16
Total: 18 KPIs (12%)
Existence of a
formal portfolio
management
process
PFM1
Portfolio
management
(PFM)
(Chiesa, Coughlan
and Voss, 1996;
Archer and
Ghasemzadeh,
1999; Beringer,
Jonas and Kock,
2013; Markham
and Lee, 2013)
3
5
Total: 8 KPIs (5%)
Commitment of
resources for
innovation/ new
products
projects
PM1
Project
management (PM)
(Cooper and
Kleinschmidt,
1995, 2007;
Adams, Bessant
and Phelps, 2006)
6
16
Total: 22 KPIs (15%)
Continuously
thinking of next-
generation
technology
TM1
Technology
management (TM)
(Prajogo and
Sohal, 2006)
4
18
Total: 22KPIs (15%)
Cross-functional
team
TEAM1
Team management
(TEAM)
(Prajogo and
Sohal, 2006;
Markham and Lee,
2013)
5
11
Total: 16 KPIs (11%)
Use of market
research tools
CO1
Commercialisation
(CO)
(Adams, Bessant
and Phelps, 2006;
Markham and Lee,
2013)
3
9
Total: 12 KPIs (8%)
Recognition of
key problems
that must be
solved with
skills that reside
outside the
organisation
IVA1
Innovation
vanguard adoption
(IVA)
(Markham and
Lee, 2013; Dubiel,
Durmusoglu and
Gloeckner, 2016;
Gurtner and
Reinhardt, 2016)
4
10
Total: 14 (10%)
Total: 146 (100%)
13
It is noteworthy that these indicators are not meant to be an end result concerning
achieving superior performance in the innovation process, but rather a means to manage
it. Proposing to apply these indicators alone, without customization and using in their
raw data formats, would not be sufficient when seeking for improved performance. The
application of these indicators works as a roadmap for companies to develop their
competences further, in terms of applying new innovation practices within the cycles
of improvements.
DISCUSSION
It is important to highlight that all of these dimensions should not be seen as
linear and sequential to the process, but they flow across the many cycles of the product
development process and consequently being more or less present in distinct stages of
the development.
In contrast to the indicators distribution, the most recurrent dimension identified
across the studies was, in fact, organisation and culture (see Table 1). Since (Pugh et
al., 1969), the structure and culture of an organisation are strongly related to the context
of the new product development functions. Further, it has been widely established that
the work environment (organisation structure and culture) makes a difference in the
level of innovation in organisations (Adams, Bessant and Phelps, 2006). The
organisation and culture dimension refers to the organisational culture within which
they work, meaning the perceived work environment, in which innovation can be
encouraged or hampered, and the way staff are organised (Cooper and Kleinschmidt,
1995; Chiesa, Coughlan and Voss, 1996; Verhaeghe and Kfir, 2002; Adams, Bessant
and Phelps, 2006; Barczak and Kahn, 2012; Nicholas et al., 2015; Lee and Markham,
2016). The corresponding example in Table 2, indicator OC1, refers to the culture based
on the concepts of teams perception concerning a shared vision; support for innovation;
participative safety, and task orientation (Anderson and West, 1996). This indicator
from the rapid assessment approach can and should be used by small manufacturing
enterprises, as it was also validated in small teams perception (Mathisen et al., 2004).
The second most cited dimension was innovation strategy. Scholars have
extensively demonstrated that activities must be consistent with an overarching
organisational strategy, implying that management must take conscious decisions
regarding innovation goals. The innovation strategy dimension represents defining and
planning a coordinated focus for the new product related processes efforts of business
units, division, product line, or an individual project. Organisations that possess
strategic leadership to enable a clear vision and to prospect future market opportunities
are considered more refined in terms of identifying a clear, new product strategy
orientation (Cooper and Kleinschmidt, 1995; Chiesa, Coughlan and Voss, 1996;
Verhaeghe and Kfir, 2002; Adams, Bessant and Phelps, 2006; Barczak and Kahn, 2012;
Markham and Lee, 2013). The indicator IS1, presented as an example in Table 2, refers
to the strategic orientation and dissemination goals for the innovation effort (e.g.,
percentage of sales, profit or growth over the next X years) (Cooper and Kleinschmidt,
1995, 2007; Tipping, Zeffren and Fusfeld, 1995). This indicator for the rapid
14
assessment presents a 5-pointed Likert scale to be compiled to be used of distinct
company sizes (Cooper and Kleinschmidt, 1995).
The third most cited dimension was team management. Team factors have been
argued as inputs for the innovation management in the new product development
process largely in literature, not only on OM research but also in the literature on
creativity. Leading organisations rely significantly on cross-functional teams
throughout the NPD process and are likely to have a individuals with potential skills to
work full time on such activities (Cooper and Kleinschmidt, 1995; Chiesa, Coughlan
and Voss, 1996; Verhaeghe and Kfir, 2002; Adams, Bessant and Phelps, 2006; Kahn et
al., 2012). Table 2 exemplified the indicator TEAM1 for the rapid assessment, which
refers to cross-functional teams (Damanpour, 1991; Prajogo and Sohal, 2006; Markham
and Lee, 2013). This indicator was also applied in small companies, especially in Asia
(Markham and Lee, 2013).
Furthermore, the knowledge management dimension is concerned with
obtaining and communicating ideas and information that underlie innovation
competencies. It includes idea generation, knowledge repository and information flows
of the new product development (Cooper and Kleinschmidt, 1995; Chiesa, Coughlan
and Voss, 1996; Verhaeghe and Kfir, 2002; Adams, Bessant and Phelps, 2006; Nicholas
et al., 2015; Lee and Markham, 2016). The indicator KM1 showed in Table 2 relates to
idea generation (Eling, Griffin and Langerak, 2016; Gurtner and Reinhardt, 2016; Lee
and Markham, 2016). This indicator for the rapid assessment was also used in small
and medium companies. It is important to highlight that there is a tendency for small
companies to present less formalized idea generation process. However, if these
companies present an innovation process in place, using formal idea generation both
for radical and incremental new product ideas leads to highest firm’s ideas success rates
(Eling, Griffin and Langerak, 2016).
The portfolio management dimension refers to the on-going review and
screening of new projects ideas, using evaluation tools, and identifying preferable
product concepts with which to proceed existing products to ensure alignment with
strategy and resource availability (Chiesa, Coughlan and Voss, 1996; Verhaeghe and
Kfir, 2002; Adams, Bessant and Phelps, 2006; Lee and Markham, 2016). The example
in Table 2, the indicator PFM1 for the rapid assessment, refers the existence of a formal
portfolio management process. This indicator was applied in small companies, and as
the previous which presents a tendency for small companies to perform a less
formalized idea generation process, it is essential to new product development (Lee and
Markham, 2016).
The technology management dimension concerns the anticipation of the
potential of new technologies, implementing long-term programs for developing
technological competences (potential), technology orientation and R&D effectiveness
when applicable (Chiesa, Coughlan and Voss, 1996; Verhaeghe and Kfir, 2002). The
illustrated example in Table 2, the indicator TM1, refers to technology potential
analysis used in the in-depth approach, with the purpose of enabling organisations to
assess in more detail their management of innovation, making possible to identify areas
15
within each where attention should be focussed. In this way, this indicator was tailored
to be used in large companies (Verhaeghe and Kfir, 2002).
The commercialization dimension includes the application of activities for
understanding customers, competitors, and macro-environmental forces in the
marketplace. Usually, more sophisticated organisations employ a variety of market
research techniques so that the customer can be involved throughout the new product
development process, testing and marketing and sales validation (Cooper and
Kleinschmidt, 1995; Adams, Bessant and Phelps, 2006; Kahn et al., 2012; Nicholas et
al., 2015). As the example in Table 2, the indicator CO1 refers to market research tools
for both small, medium and large companies (Nicholas et al., 2015) and should be used
within the scope of the rapid assessment.
The innovation vanguard adoption dimension is concerned with the new trends
in innovation and new product development practices: openness, meaning development
regarding external and internal collaboration (Adams, Bessant and Phelps, 2006;
Nicholas et al., 2015; Lee and Markham, 2016); sustainability orientation referring to
the incorporation of the triple bottom line into the new product development process
(Verhaeghe and Kfir, 2002; Lee and Markham, 2016); and, servitization as an
intentional and coordinated effort to incorporate Product Service Systems (PSS) (Lee
and Markham, 2016). The indicator CO1 for the rapid assessment presented in Table 2
refers to openness as the recognition of key problems that must be solved with skills
that reside outside the organisation. This indicator was used in companies with various
sizes, predominantly small firms (Dubiel, Durmusoglu and Gloeckner, 2016).
It is important to highlight that the vast majority the use of KPIs, and
consequently, performance measurement systems, were designed and tested in, and for,
large companies (Hudson Smith and Smith, 2007). Literature has demonstrated that
there can be considerable difficulties in implementing these indicators and systems
effectively and that difficulties are particularly prevalent in smaller companies
(McAdam, McConvery and Amstrong, 2004). The basis for the development of a
performance measurement also focused on small and medium firms is grounded on an
incremental approach (Hudson Smith and Smith, 2007). This progressive approach
aims to design performance indicators one at a time, according to current strategic
priorities and immediately cascade the measurements down to the operational level, to
ensure implementation. In this way, the present dimensions aim to give a rapid
assessment to provide a diagnostics and, in time, works as a checklist providing
indicators which can be selected, defined and implemented, even one at a time,
respecting the strategic priorities of the company.
Overall, although the literature present notable approaches regarding
performance measurement and categorisation of dimension and a few KPIs to the
innovation process, most approaches do not provide a comprehensive database of
indicators where managers can select from. In this sense, the results of this research
contribute to the enhancement of knowledge concerning: the systematic identification
of existing KPIs; systematization of the KPIs into relevant classification to the
innovation process; creation the database and their related indicator’s attributes.
16
CONCLUSION
This paper reported the definition and systematization of a set of KPIs for
measuring the innovation process. The research method was designed on the grounds
of a systematic literature review to cover performance measurement, accompanied by
a systematization of the KPIs. The resulting 146 KPIs were catalogued and
systematized in an electronic spreadsheet, being 34 for rapid assessment and suitable
for small-sized companies and 112 for the in-depth approach. This database in the
current form is not intended to offer specific support for selecting and defining the
performance indicators without assuming contextualization (designing specifically to
meet local needs of the company), avoiding “one size fits all” type of framework. It
constitutes an additional resource advancing the knowledge for both academics and
practitioners working with performance measurement in the innovation process.
The main findings of this paper are: i) the identification of nine dimensions:
innovation strategy; knowledge management; organisation and culture; portfolio
management; project management; technology management; team management;
commercialization and innovation vanguard adoption referring to openness,
sustainability and servitization orientation; ii) the identification of most cited
dimensions: organisation and culture; innovation strategy and team management and
iii) the identification of highest numbers of KPIs: project management and technology
management, which are not correspondent to the most cited dimensions, probably due
to the role played by of PMI and the evolvement of literature, respectively.
Furthermore, from an applied perspective, the proposed study also enables a
benchmarking of process-related KPIs based on the state of the art and in the content
of the database.
A few limitations of this research can also be pointed out. First of all, the KPI
database was systematized based on purely academic sources, journals without
considering a potential systematic review of “state-of- the-practice” sources and
databases. Secondly, the classification and systematization of the catalogued KPIs are
subjected to the researchers’ own judgements regarding the classification of two-level
analysis. Thirdly, due to the abovementioned judgement analysis, their interpretation
of the KPIs was sometimes solely based on their titles, and surrounding definitions and
further assumptions were made in a subjectively.
These limitations can be addressed by the following: i) broadening the scope of
the literature review to cover practitioner-oriented sources to include insights from
practice, which could potentially lead to a higher number of KPIs systematized and ii)
subject the KPIs systematization to a panel of experts in the fields of innovation
management, product development and performance measurement.
To effectively capture and measure the performance of the innovation in the
new product development process, more efforts should be put on developing a step-by-
step procedure showing how to deploy and customize this KPIs to a practical
application within an organisation. Hence, future research should overcome potential
limitations of this research, in addition, to propose actions to be taken within the overall
context of this paper. Next steps within the frame of this research include addressing
17
the limitations by extending the scope of the systematic literature and submitting the
KPIs database systematisation to an expert panel.
Finally, as mentioned before, this paper is part of a more significant research to
develop a performance measurement framework. The resulting measurement
framework will not only contribute to fulfilling a gap in literature but also provide a
useful basis for managers in practice to select KPIs to monitor their innovation
processes, diagnose limitations and prescribe appropriate actions. In the future, this
prescriptive support tool should be instantiated with empirical data by conducting an
action research and geared towards laying out the fundamental rationale of how
performance measurement framework can improve an organisation’s innovation
performance.
ACKNOWLEDGEMENT
This paper is part of a greater Ph.D. research that proposes a performance framework
for the innovation process. Sincere thanks to the Department of Mechanical &
Manufacturing Engineering of Trinity College Dublin who funded this research.
REFERENCES
Acosta, C. L. M., Araújo, c. s. and Trabassos, L. G. (2002) ‘A Review of product
development performance metrics investigations with emphasis on the designer level’,
in International Design Conference - Design 2002. Dubrovnik, pp. 621–626.
Adams, R., Bessant, J. and Phelps, R. (2006) ‘Innovation management measurement: a
review’, International Journal of Management Reviews, 8(1), pp. 21–47. doi:
10.1111/j.1468-2370.2006.00119.x.
Adriaanse, L. S. and Rensleigh, C. (2013) ‘Web of Science, Scopus and Google
Scholar: A content comprehensiveness comparison’, The Electronic Library, 31(6), pp.
727–744. doi: 10.1108/EL-12-2011-0174.
Anderson, N. and West, M. (1996) ‘The team climate inventory: development of the
TCI and its applications in teambuilding for innovativeness’, European Journal of
Work and Organizational Psychology, 5(1), pp. 53–66. doi:
10.1080/13594329608414840.
Archer, N. and Ghasemzadeh, F. (1999) ‘An integrated framework for project portfolio
selection’, International Journal of Project Management, 17(4), pp. 207–216. doi:
10.1016/S0263-7863(98)00032-5.
Baines, T. and W. Lightfoot, H. (2013) Servitization of the manufacturing firm:
Exploring the operations practices and technologies that deliver advanced services,
International Journal of Operations & Production Management. doi: 10.1108/IJOPM-
02-2012-0086.
Barczak, G. and Kahn, K. B. (2012) ‘Identifying new product development best
practice’, Business Horizons, 55(3), pp. 293–305. doi: 10.1016/j.bushor.2012.01.006.
Berg, P., Pihlajamaa, J., Poskela, J., Lempiälä, T., Haner, U. and Mabogunje, A. (2008)
‘Measurement of the innovation front end: Viewpoint of process, social environment
and physical environment’, in PICMET: Portland International Center for
Management of Engineering and Technology, Proceedings, pp. 1112–1120. doi:
18
10.1109/PICMET.2008.4599721.
Beringer, C., Jonas, D. and Kock, A. (2013) ‘Behavior of internal stakeholders in
project portfolio management and its impact on success’, International Journal of
Project Management, 31, pp. 830–846. doi: 10.1016/j.ijproman.2012.11.006.
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M. and Khalil, M. (2007)
‘Lessons from applying the systematic literature review process within the software
engineering domain’, Journal of Systems and Software. Elsevier Inc., 80(4), pp. 571–
583. doi: 10.1016/j.jss.2006.07.009.
Calik, E. and Bardudeen, F. (2016) ‘A Measurement Scale to Evaluate Sustainable
Innovation Performance in Manufacturing Organizations’, Procedia CIRP. Elsevier
B.V., 40, pp. 449–454. doi: 10.1016/j.procir.2016.01.091.
Chiesa, V., Coughlan, P. and Voss, C. A. (1996) ‘Development of a technical
innovation audit’, Journal of Product Innovation Management, 13, pp. 105–136.
Available at: http://onlinelibrary.wiley.com/doi/10.1111/1540-5885.1320105/pdf.
Cooper, R. G. and Kleinschmidt, E. J. (1995) ‘Benchmarking the firm’s critical success
factors in new product development’, Journal of Product Innovation Management,
12(5), pp. 374–391. doi: 10.1016/0737-6782(95)00059-3.
Cooper, R. G. and Kleinschmidt, E. J. (2007) ‘Winning businesses in product
development: the critical success factors’, Research-Technology Management, May-
June, pp. 1–15. doi: 10.1080/08956308.2007.11657441.
Cordero, R. (1990) ‘The measurement of innovation performance in the firm: An
overview’, Research Policy, 19(2), pp. 185–192. doi: 10.1016/0048-7333(90)90048-B.
Crawford, C. M. and Di Benedetto, C. A. (2011) New products management. Tenth,
Journal of Engineering and Technology Management. Tenth. New York: McGraw-Hill
Irwin. doi: 10.1016/0923-4748(93)90075-T.
Crossan, M. M. and Apaydin, M. (2010) ‘A multi-dimensional framework of
organizational innovation: a systematic review of the literature’, Journal of
Management Studies, 47(6), pp. 1154–1191. doi: 10.1111/j.1467-6486.2009.00880.x.
Czuchry, A. J. and Yasin, M. M. (2001) ‘Enhancing global competitiness of small and
mid sized firms: A rapid assessment methodology approach’, Advances in
Competitiveness Research, 9(1), pp. 87–99.
Damanpour, F. (1991) ‘Organizational Innovation: A Meta-Analysis of Effects of
Determinants and Moderators’, Academy of Management Journal, 34(3), pp. 555–590.
doi: 10.2307/256406.
Dubiel, A., Durmusoglu, S. S. and Gloeckner, S. (2016) ‘Firm characteristics and NPD
program success: the significant influence of global discovery management’, Journal
of Product Innovation Management, 33(S1), pp. 86–100. doi: 10.1111/jpim.12330.
EC (2016) Science, Research and Innovation performance of the EU: A contribution to
the open innovation, open science, open to the world agenda, European Commission,
DG for Research and Innovation. Luxembourg. doi: 10.2777/427046.
Eling, K., Griffin, A. and Langerak, F. (2016) ‘Consistency matters in formally
selecting incremental and radical new product ideas for advancement’, Journal of
Product Innovation Management, 33(S1), pp. 20–33. doi: 10.1111/jpim.12320.
19
Erkens, M., Wosch, S., Piller, F. and Lüttgens, D. (2014) Measuring open innovation:
a toolkit for succesful innovation teams, Performance by EY.
Glaser, B. G. and Strauss, A. L. (1967) The Discovery of Grounded Theory: Strategies
for Qualitative Research, Observations. doi: 10.2307/2575405.
Gurtner, S. and Reinhardt, R. (2016) ‘Ambidextrous idea generation-antecedents and
outcomes’, Journal of Product Innovation Management, 33(S1), pp. 34–54. doi:
10.1111/jpim.12353.
Hudson Smith, M. and Smith, D. (2007) ‘Implementing strategically aligned
performance measurement in small firms’, International Journal of Production
Economics, 106(2), pp. 393–408. doi: 10.1016/j.ijpe.2006.07.011.
Issa, I. I., Pigosso, D. C. A., McAloone, T. C. and Rozenfeld, H. (2015) ‘Leading
product-related environmental performance indicators: a selection guide and database’,
Journal of Cleaner Production. Elsevier Ltd, 108, pp. 321–330. doi:
10.1016/j.jclepro.2015.06.088.
Kahn, K. B., Barczak, G., Nicholas, J., Ledwith, A. and Perks, H. (2012) ‘An
examination of new product development best practice’, Journal of Product Innovation
Management, 29(2), pp. 180–192. doi: 10.1111/j.1540-5885.2011.00888.x.
Kaplan, R. S. and Norton, D. P. (1992) ‘The balanced scorecard –measures that drive
performance’, Harvard Business Review, ((January–February)), pp. 71–79. doi:
10.1017/CBO9781107415324.004.
Katila, R., Chen, E. L. and Piezunka, H. (2012) ‘All the right moves: how
entrepreneurial firms compete effectively’, Strategic Entrepreneurship Journal, 6, pp.
116–132. doi: 10.1002/sej1130.
Kleinknecht, A. (1987) ‘Measuring R&D in Small Firms: How Much are we Missing?’,
The Journal of Industrial Economics, 36(2), pp. 253–256. Available at:
http://www.jstor.org/stable/2098417.
Lee, H. and Markham, S. K. (2016) ‘PDMA Comparative performance assessment
study (CPAS): methods and future research directions’, Journal of Product Innovation
Management, 33(S1), pp. 3–19. doi: 10.1111/jpim.12358.
Markham, S. K. and Lee, H. (2013) ‘Product development and management
association’s 2012 comparative performance assessment study’, Journal of Product
Innovation Management, 30(3), pp. 408–429. doi: 10.1111/jpim.12025.
Mascarenhas Hornos da Costa, J., Oehmen, J., Rebentisch, E. and Nightingale, D.
(2014) ‘Toward a better comprehension of Lean metrics for research and product
development management’, R&D Management, 44(4), pp. 370–383. doi:
10.1111/radm.12074.
Mathisen, G. E., Einarsen, S., Jørstad, K. and Brønnick, K. S. (2004) ‘Climate for work
group creativity and innovation: Norwegian validation of the team climate inventory
(TCI)’, Scandinavian Journal of Psychology, 45(5), pp. 383–392. doi: 10.1111/j.1467-
9450.2004.00420.x.
McAdam, R., McConvery, T. and Amstrong, G. (2004) ‘Barriers to innovation within
small firms in a peripheral location’, International Journal of Entrepreneurial
Behaviour & Research, 10(3), pp. 206–221. doi: http://dx.doi.org/10.1108/MRR-09-
2015-0216.
20
Neely, A., Gregory, M. and Platts, K. (1995) ‘A Literature Review and Research
Agenda’, International Journal of Operation Management & Production Management,
15(4), pp. 80–116. doi: 10.1108/01443579510083622.
Nicholas, J., Ledwith, A., Aloini, D., Martini, A. and Nosella, A. (2015) ‘Searching for
radical new product ideas: exploratory and confirmatory factor analysis for construct
validation’, International Journal of Technology Management, 68(1–2), pp. 70–98. doi:
10.1504/IJTM.2015.068779.
Nilsson, S. and Ritzén, S. (2014) ‘Exploring the use of innovation performance
measurement to build innovation capability in a medical device company’, Creativity
and Innovation Management, 23(2), pp. 183–198. doi: 10.1111/caim.12054.
OECD (2005) Oslo Manual - Guidelines for collecting and interpreting innovation
data. 3rd edn, Communities. 3rd edn. Paris. doi: 10.1787/9789264013100-en.
Prajogo, D. I. and Sohal, A. S. (2006) ‘The integration of TQM and technology/R&D
management in determining quality and innovation performance’, Omega, 34(3), pp.
296–312. doi: 10.1016/j.omega.2004.11.004.
Pugh, D. S., Hickson, D. J., Hinings, C. R. and Turner, C. (1969) ‘The context of
organization structures’, Administrative Science Quarterly, 14(1), pp. 91–114.
Rodrigues, V. P., Pigosso, D. C. A. and McAloone, T. C. (2016) ‘Process-related key
performance indicators for measuring sustainability performance of ecodesign
implementation into product development’, Journal of Cleaner Production. Elsevier
Ltd, 139, pp. 416–428. doi: 10.1016/j.jclepro.2016.08.046.
Shaner, M. B., Beeler, L. and Noble, C. H. (2016) ‘Do We Have to Get Along to
Innovate? The Influence of Multilevel Social Cohesion on New Product and New
Service Development’, Journal of Product Innovation Management, 33, pp. 148–165.
doi: 10.1111/jpim.12327.
Sudarsan, R., Fenves, S. J., Sriram, R. D. and Wang, F. (2005) ‘A product information
modeling framework for product lifecycle management’, CAD Computer Aided
Design, 37(13), pp. 1399–1411. doi: 10.1016/j.cad.2005.02.010.
Tipping, J. W., Zeffren, E. and Fusfeld, A. R. (1995) ‘Assessing the value of your
technology’, Research Technology Management, 38, pp. 22–39.
Verhaeghe, A. and Kfir, R. (2002) ‘Managing innovation in a knowledge intensive
technology organisation (KITO)’, R&D Management, 32(5), pp. 409–417. doi:
10.1111/1467-9310.00272.
Zizlavsky, O. (2015) ‘Approaches to Innovation Process Assessment: Complex Results
from an Exploratory Investigation’, International Journal of Engineering Business
Management, 7, pp. 1–16. doi: 10.5772/62052.
