On reviewing results of design research

Abstract

Design research is emerging research approach. Concerning the most important research outcome of design research, an artefact or innovation, the business needs are then emphasized. In their seminal article of design research, March and Smith (1995) stressed on some universal criteria. Later Hevner et al. (2004) presented their 7 guidelines. In this paper their sketched approaches are further elaborated. Two new resource types (human and informational ones) are proposed and the goal function is defined for reviewing goodness of design research. In the same connection scientific research results are classified into three classes in general, and into four classes in design research especially. Both classifications are novel. The extensions and results are then discussed.

Full text

ON REVIEWING OF RESULTS IN DESIGN RESEARCH
Pertti Järvinen
University of Tampere, Finland
Department of Computer Sciences
pj@cs.uta.fi
Abstract
Design research is emerging research approach. Concerning the most important research outcome of
design research, an artefact or innovation, the business needs are then emphasized. In their seminal
article of design research, March and Smith (1995) stressed on some universal criteria. Later Hevner
et al. (2004) presented their 7 guidelines. In this paper their sketched approaches are further
elaborated. Two new resource types (human and informational ones) are proposed and the goal
function is defined for reviewing goodness of design research. In the same connection scientific
research results are classified into three classes in general, and into four classes in design research
especially. Both classifications are novel. The extensions and results are then discussed.
Keywords: Design research, Research outcomes, Reviewing, Goal function.
1388

1 INTRODUCTION
Hevner et al. (2004) inform the community of IS (information systems) researchers and practitioners
of how to conduct, evaluate, and present design science research. They do so “by describing the
boundaries of design-science within the IS discipline via a conceptual framework for understanding
information systems research and by developing a set of guidelines for conducting and evaluating
good design-science research”. Referring to March and Smith (1995) Hevner et al. (2004) indirectly
define IT artifact: “We include not only instantiations in our definition of the IT (information
technology) artifact but also the constructs, models, and methods applied in the development and use
of information systems. We do not include people or elements of organizations in our definition nor do
we explicitly include the process by which such artifacts evolve over time”. They exclude people from
their artifact. Another resource type that also seems to be excluded is informational resource by which
we mean both data, information and knowledge. In this paper we shall analyze some consequences
that inclusion of the informational and human resources may create.
According to Hevner et al. (2004) the goal of design-science research is utility. This goal does not
seem to cover everything. Van der Heijden (2004) studied the differences in user acceptance models
for productivity-oriented (or utilitarian) and pleasure-oriented (or hedonic) information systems.
Hedonic information systems aim to provide self-fulfilling rather than instrumental value to the user,
are strongly connected to home and leisure activities, focus on the fun-aspect of using information
systems, and encourage prolonged rather than productive use. This counterexample encourages us to
find another expression for the goal of design-science research in this paper.
Hevner et al. (2004) recommend that the researcher should report three contributions: the design
artefact, its construction and evaluation processes. For those processes they give the following
rationale: “Because design-science artefacts are often the ‘machine’ part of the human-machine system
constituting an information system, it is imperative to understand why an artefact works or does not
work to enable new artefacts to be constructed that exploit the former and avoid the latter”. The latter
alternative, the artefact does not work, is explicitly mentioned should be hence included into research
contributions.
Design science is fundamentally a problem-solving paradigm. “As technical knowledge grows, IT is
applied to application areas, that were not previously believed to be amenable to IT support” (Markus
et al. 2002). Hevner et al. (2004) write that “technological advances are the result of innovative,
creative design science processes. If not capricious, they are at least arbitrary (Brooks 1987) with
respect to business needs and existing knowledge.” The latter is an example how the design science
paradigm should be extended with an opportunity, i.e. advances in technical, human (organizational)
and informational resources offer an opportunity to build a new innovative artefact.
To summarize, we shall in this paper make some extensions in relation to results in design research.
First, in addition to the IT resource we also consider human and informational resources (Section 2).
Secondly, based on differing goals and different interested parties we analyse goal functions in design
research (Section 3). Thirdly, both considerations have a certain influence on our classification of
results of design research (Section 4). Finally, we discuss about implications and limitations of our
outcomes. Our research approach can be called as a conceptual-analytical one (Järvinen 2004, Chapter
2).
1389

2 ON HUMAN AND INFORMATIONAL RESOURCES
In this section we first analyse some connection between people-based and technology-based artefacts
and some properties of people, and thereafter we consider traits of informational resources. Hevner et
al. (2004) argue that “a combination of technology-based artefacts (e.g., system conceptualizations and
representations, practices, technical capabilities, interfaces, etc.) organization-based artefacts (e.g.,
structures, compensation, reporting relationships, social systems, etc.), and people-based artefacts
(e.g., training, consensus building, etc.) are necessary to address issues concerning the acceptance of
information technology in organizations”. Hence Hevner et al. know that in practice there are other
artefacts than the IT artefacts on which they only liked to concentrate. Those other artefacts concern
human resources paying attention to people as such and organizational structures.
Referring to Henderson and Venkatraman (1993) Hevner et al. (2004) state that “the effective
transition of strategy into infrastructure requires extensive design activity on organizational design to
create an effective organizational infrastructure and information systems design activity to create an
effective information system infrastructure. These are interdependent design activities that are central
to IS discipline.” Although such tight connections exist between those two activities Hevner et al.
limited their discussion of design science to activities of building the IS infrastructure within the
business organization.
We understand their decision when we look at Aulin’s (1989) taxonomy of dynamic systems. The
regularly behaving IT artifact seems to belong to the class of nilpotent systems where a system has a
certain rest point, but a human being to the class of self-steering systems where a system has the
infinite goal function in time. The self-steering system as a model of human being can represent such a
human property as free will, i.e. the self-steering system can change its goal function whenever it
wants. Another property of the self-steering system is that the same state never returns which also
holds with a human being. This fact means that people, in principle, are non-manipulable research
objects, but researchers can to a certain extent manipulate IT technology in their experiments.
Inclusion both IT and people-artifacts may have a great influence on selection of research approach.
Levitin and T.C. Redman (1998), compared the data resource with the traditional resources (financial,
human, plant, equipment and raw materials) and demonstrated its importance. Knowledge as a
resource was long recognized especially in artificial intelligence. When those types of informational
resources are stored in data and knowledge bases, they can be utilized by IT artefacts. They can play a
central role in business as Christiaanse and Venkatraman (2002) demonstrated in relations between a
airline company (American Airlines, AA) and travel agencies. The computerized reservation system
brought a lot of revenues to the AA, because they were the first entrants to the field.
One reason that Hevner et al. (2004) “forget” the informational resource type could be that they
assumed informational resources to be a inseparable part of IT, because data and knowledge bases
were stored in computers. Benbasat (2003) informed us that they (Benbasat and Zmud 2003) left
informational resources out of their nomological network because of the close connection of the stored
data and computers.
Although data and knowledge stored in computers behave regularly, and hence those kinds of
informational resources resemble IT and form an important component of the organizational memory,
the latter still has two other components: experts and prototypes. Information about existing solutions
1390

resides within the artefacts (prototypes) themselves (Hargadon and Sutton 1997) and those prototypes
can be any technical tools and equipments, not only IT artefacts.
Experts as human resources have special capabilities. They can compete with IT artefacts as a memory
and processor of information. One key question in building a certain IT artefact is how to distribute the
tasks between a user and computer. Concerning human capabilities experts can have different types of
knowledge. Blackler (1995) defines the five knowledge types as follows: Embrained knowledge,
called ’knowledge that’ or ‘knowledge about’, is knowledge that is dependent on conceptual skills and
cognitive abilities. Embodied knowledge, called ’knowledge how’ or ’knowledge of acquaintance’, is
action oriented and is likely to be only partly explicit. Encultured knowledge refers to the processes of
achieving shared understandings. Embedded knowledge resides in systemic routines. Encoded
knowledge is information conveyed by signs and symbols. The first four knowledge types are typical
for people, the IT artefacts and people both can have the encoded knowledge. This difference can
partly explain why Hevner et al. (2004) excluded both the human and informational resources from
their consideration.
There is also another study demonstrating problematic relationships between information and people.
Constant et al. (1994) showed that as technology for information access improves, people have more
opportunities to share information. They developed the model for information sharing and the results
of three experiments on attitudes about sharing technical work and expertise in organizations reported
that sharing tangible information work (computer program) may depend on prosocial attitudes and
norms of organizational ownership, and sharing expertise may depend on people's own self-expressive
needs.
We in this section showed that people as self-steering systems which much differ from IT artefacts. To
a certain extent they are competitors with computers in storing and processing information. In
addition, people can have different types of knowledge than IT artefacts which can only use encoded
data and knowledge. Other materialized artefacts, too, can have knowledge which can be difficult to
read (Haythornthwaite 2006).
3 ON DIFFERENT GOAL FUNCTIONS IN DESIGN RESEARCH
In this section we demonstrate that the domain of the IT artefact is restricted concerning evaluation of
utility of the IT artefact. Hence, it should be extended. We also pay attention to different stake holders
in connection with IT artefacts, and to other criteria than utility.
Hevner et al start their article with phrase: “Information systems are implemented within an
organization for the purpose of improving the effectiveness and efficiency of that organization”. This
expression emphasizes the organization as a suitable domain for evaluating consequences, e.g.,
effectiveness, of the IT artefact. In the previous section we cited organization-based artefacts that can
be related with badly paid users having influence on the effectiveness of the IT artefact. The similar
result can be achieved, if users are not relevantly trained (cf. people-based artefact). Hence, it is at
least debatable to discuss about extension of the evaluation range from the IT artefact to the domain
which also include its users.
1391

Hevner et al. (2004) demanded that “the utility, quality and efficacy of a design artefact must be
rigorously demonstrated via well-executed evaluation methods”. They also expressed their as follows:
“IT artefacts can be evaluated in terms of functionality, accuracy, performance, reliability, usability, fit
with the organization, and other relevant quality attributes”. Expression ‘relevant quality attributes’ is
skilfully selected, because quality is a multidimensional concept at least having four dimensions
(Reeves and Bednar 1994): I. Excellence, II. Value, III. Conformance to specifications and IV.
Meeting and/or exceeding customers’ expectations. The first one is the general dimension. Different
interested parties can be connected with three others as follows: II. Value (managers), III.
Conformance to specifications (systems analysts) and IV. Meeting and/or exceeding customers’
expectations (customers). When Hevner et al. (2004) emphasize business needs they at the same
moment select managers’ view on quality. But systems analysts’ and customers’ view is then
forgotten. In connection with e-commerce Chua et al. (2005) presented a stakeholder theory where
they saw the following stakeholders: Customers, managers, employees, suppliers, indirect
stakeholders, investors, and regulators. To our mind, all the most important stakeholders must be taken
into account when the outcomes of design research are evaluated.
There is another reason why expression ‘business needs’ is not the best one. Hevner et al. (2004)
wrote that “design-science research addresses important unsolved problems in unique or innovative
ways or solved problems in more effective or efficient ways”. They, thus, (over)emphasize problems
as important initiators for design-research and neglect new opportunities which various resources
offer. The innovation based on a certain new opportunity does not necessarily have any connection
with business needs at the moment when it is created. Later the opportunity-based innovation can have
a great economic value.
According to Hevner et al. (2004) “in design science, computational and mathematical methods are
primarily used to evaluate the quality and effectiveness of artefacts. Sometimes the optimal solution
could not proved, but sometimes heuristics should be accepted, and then “the use of heuristics to find
“good” design solutions opens the question of how goodness is measured”. His citation demonstrates
that Hevner et al. also thought other measures than utility. It was far-sighted, because other reasons
than utility emerged, e.g. pleasure-oriented one (Van der Heijden 2004). When Iivari (2007) proposes
that the some information systems could produce aesthetic influences, Hevner et al. (2004) required
that “design evaluation should include an assessment of the artefact’s style”. Iivari also have other new
innovative categories of services (e.g., to accompany and to phish) which information systems produce
and whose role or function cannot be called utility.
Hence, term ‘utility’ does not cover all the roles or functions which services of information systems
have. We need a new, more general term to describe the goals of information systems, and we propose
that it could be the goal function under which all kinds of different interests can be collected. We shall
in the next section use it in our classification of design research outcomes.
4 THE CLASSIFICATION OF RESEARCH RESULTS IN DESIGN
RESEARCH
According to March and Smith (1995) all the artefacts built in the design research projects are not
considered valuable to be published. They only accept a totally new artefact or a significant
improvement in the earlier one. Van Aken (2004) recommends supplementing those categories with
1392

unsuccessful projects, too. He might like to prevent other researchers to make the same or similar
mistakes or give them a chance to improve old solutions or to solve unsolvable problems.
In order to know which artefact is new, significantly better or unsuccessful, we must compare the
result of design research with those reported in the literature. In her case study article Eisenhardt
(1989) calls this 7th step: Enfolding literature. It means two tasks: 1) Comparison with conflicting
literature and 2) Comparison with similar literature. Eisenhardt continues: An essential feature of
theory creating is comparison of the emergent concepts, theory or hypotheses with extant literature.
This involves asking what is similar to, what does it contradict and why. A key to this process is to
consider a broad range of literature. Examining literature, which conflicts with the emergent theory, is
important for two reasons. First, if researchers ignore conflicting findings, then confidence in the
findings are reduced. For example, readers may assume that the results are incorrect (a challenge to
internal validity), or if correct, are idiosyncratic to the specific cases of the study (a challenge to
generalizability). Second and perhaps more importantly, conflicting literature represents an
opportunity. The juxtaposition of conflicting results forces researchers into a more creative,
framebreaking mode of thinking than they might otherwise be able to achieve. The result can be
deeper insight into both the emergent theory and the conflicting literature, as well as sharpening of the
limits to generalizability of the focal research. – Literature discussing similar findings is important as
well because it ties together underlying similarities in phenomena normally not associated with each
other. The result is often a theory with stronger internal validity, wider generalizability, and higher
conceptual level.
We interpret Eisenhardt’s results as follows: If our results are similar to in the literature, then our
results support the earlier results. If our results are dissimilar to ones in the literature then there are two
possibilities. First, our results are contradictory to the earlier results. The latter must then be falsified.
Secondly, our results differ from the earlier results in such a way that they are not comparable with
them. Hence they are totally new. As a conclusion we have derived the classification of the
implications of a study to the theory (Table 1): D1.1 Totally new results, D1.2 The similar results to
the earlier ones and D1.3 The conflicting results compared with the earlier ones. The latter often have
supplemented with various speculations about reasons or potential explanations of conflicting results.
This classification concerns results in natural and behavioural sciences.
Code Meaning
D1.1
D1.2
D1.3
The totally new results
The similar results to the earlier ones
The conflicting results compared with the earlier ones
Table 1. Classification of research results in natural and behavioural sciences
In order to relate our classification to design research we must take one aspect of design research into
account, namely that the purpose of the innovation or artefact has to have a certain goal function to
demonstrate and measure how good our result is. We can now formulate our classification of research
results in design research as follows: D1.1 The totally new innovation or artefact, D1.2 our innovation
or artefact is similar to some earlier innovation or artefact and it produces the equal value of goal
function as some earlier innovation, and D1.3 our innovation or artefact is different from some earlier
innovation or artefact and it produces the better or worse value of goal function compared with the
best earlier one. Unsuccessful building processes where the building project failed clearly need their
own category D1.4.
1393

The goal function in classes D1.2 and D1.3 requires more elaboration. The goal function contains the
values the stakeholders expressed in their desires concerning the new artefact or innovation built.
Those values can concern different factors: financial, aesthetic, enjoyable (Van der Heijden 2004,
Iivari 2007)), etc. ones. We assume that all the value factors can be measured or estimated. Class
D1.2, the values of the new artefact or innovation and of the best earlier one are equal, is a matter of
taste. The knowledge repository already has one artefact or innovation with the equal value of the goal
function, why should we have another one?
In case D1.3 (differing goal functions) both scientific and practical aspects emphasize the selection of
the better artefact or innovation, not the worse one. (Van Aken (2006) proposed that the worse ones
could be classified into category D1.4.) We therefore call the better alternative D1.3b. We can now
present our classification for design research results in the Table 2.
Code Meaning
D1.1
D1.2
D1.3b
D1.4
The totally new artefact or innovation
The value of the goal function associated with the new artefact or innovation is equal as the
value of the best earlier innovation
The value of the goal function associated with the new artefact or innovation is better than the
value of the best earlier innovation
The building project failed
Table 2. Classification of design research results
5 DISCUSSION
In this paper we augmented Eisenhardt’s (1989) differentiation between 1) Comparison with
conflicting literature and 2) Comparison by taking the totally new results as the third separate class.
Thereafter we tried to apply our classification to design research, and we achieved the classification:
D1.1 The totally new artefact or innovation, D1.2 The value of the goal function associated with the
new artefact or innovation is equal as the value of the best earlier innovation, D1.3b The value of the
goal function associated with the new artefact or innovation is better than the value of the best earlier
innovation, and D1.4 The building project failed. The “similar” case (D1.2) seemed to be curiosity.
Our classification for design research results differs from the classification for natural and behavioural
sciences results because of the essential role the goal function plays in design research. Both our
classification in Table 1 and our classification in Table 2 seem to be the new ones, and hence they
must be included into knowledge repository (Hevner et al. 2004).
Based on our results we recommend that a researcher should in the future show that his/her
contribution in design research is either 1) the totally new artefact or innovation, or 3b) the value of
the goal function associated with the new artefact or innovation is better than the value of the best
earlier innovation, or 4) his/her project failed. We prefer ‘proof by demonstration’ than ‘by design’.
Motivation both from science and practice should be seen in Introduction, and implications to science
and practice in Discussion. The same recommendations are applicable to work of a referee (cf.
Parberry 1989 and Smith 1990) and a critical practitioner.
1394

We already in the text above related our consideration to Hevner et al. (2004). Hence, it is necessary to
evaluate whether our proposals change the 7 guidelines presented by Hevner et al. or not. Guideline 1
(Design as an artefact) must be slightly adjusted, because we extended IT resources with both human
and informational resources. We could call this guideline as “Design as an Innovation”. To our mind,
Guidelines 2 (Problem Relevance) and 6 (Design as a Search Process) are too much concentrated in
“problem” and to this end “opportunity” could be more brought out. Guideline 3 (Design Evaluation)
could be adjusted by also accepting other major stakeholders with potentially differing goals than
managers. Guideline 4 (Research Contributions) could be extended by Table 2 and the goal function is
a bit more general expression than utility only. Guideline 5 (Research Rigor) does not need much
adjustment. For certainty, we refer to March and Smith (1995) and like to emphasize that their
proposal, ‘fidelity with real world phenomena’, is a very good criterion for the model of a
human being. Guideline 7 (Communication of Research) does not need any adjustment.
Our proposal for extension that human and informational resources should be included into the artefact
is important because of many reasons. First, goodness of the IT artefact depends on its users.
Secondly, data and knowledge play often a more central role in information systems than technology.
Thirdly, in building an information system, for a certain task there are two alternative ‘performers’,
either people or IT technology which must be considered together. Fourthly, the introduction of the
new IT artefact usually causes changes in jobs by eliminating and changing the old tasks and by
inserting some new ones; and this fact once again emphasizes the wider view on the innovation than
the IT technical one only.
Van Aken (2004) recommends that we do not only include instantiations and its derivatives as such
into the knowledge repository of design research but also the general solution concept for a type of
field problem. Our own interpretation is that the general solution concept is based on some new
characteristics of available a) technical, b) human and/ or c) informational resources or their
combination. The emphasis of different resources needed in building, usage and maintenance of the
new artefact or innovation may help to calculate or estimate different value factors. Both those
proposals require more research, because their connections or associations with a goal function are not
immediately clear.
In Section we showed that the introduction of the new IT artefact usually causes some organizational
changes. Buchanan (2003) found “the methodological implications arising from competing narratives
of an organizational change process in a large acute city teaching hospital. This qualitative case study
was informed by a processual-contextual perspective, and relied on an interpretive constructivist
epistemology. Two forms of contradiction are revealed. First, differing accounts were offered of
substantive dimensions of the change programme. Second, the impact of change on organizational
effectiveness was indeterminate. This study suggests that the unitary, authentic narrative is illusory.
Political motivations underpinning account-giving, and phenomenological variations in the lived
experience of change, make competing narratives a naturally occurring phenomenon, not a
methodological aberration.” These findings lead the following reasoning: The same changes can be
seen in different ways. In an organization there can already before be or the introduction of the new IT
artefact can lead to conflicting groups with differing goal functions. Hence, it is necessary to further
study people’s role in connection with technological changes.
In two in-depth case studies Doherty et al. (2006) empirically studied the role of interpretive
flexibility. Interpretive flexibility, which can be defined as “the capacity of a specific technology to
sustain divergent opinions” (Sahay and Robey 1996), means that different people can see the same
1395

technology in different ways. This is a significant finding and requires to even consider a pure IT
artefact in a new way in our prospective research settings.
References
Aulin A. (1989). Foundations of Mathematical System Dynamics: The Fundamental Theory of Causal
Recursion and its Application to Social Science and Economics. Pergamon Press, Oxford.
Benbasat I. (2003), private communication.
Benbasat I. and Zmud R.W. (2003). The identity crisis within the IS discipline: Defining and
communicating the discipline’s core properties. MIS Quarterly, 27 (2), 183-194.
Blackler, F. (1995). Knowledge, knowledge work and organizations: An overview and interpretation.
Organization Studies, 16 (6), 1021-1046.
Brooks F.P. (1987). No silver bullet: Essence and accidents of software engineering. IEEE Computer,
20 (4), 10-19.
Buchanan D.A. (2003). Getting the story straight: Illusions and delusions in the organizational change
process. Tamara Journal of Critical Postmodern Organization Science, 2 (4), 7-21.
Christiaanse E. and Venkatraman N. (2002). Beyond SABRE: An empirical test of expertise
exploitation in electronic channels. MIS Quarterly, 26 (1), 15-38.
Chua C.E.H., Khoo, H.M., Straub D.W. and Kadiyala S. (2005). The evolution of e-Commerce
research: A stakeholder perspective. Journal of Electronic Commerce Research, 6 (4), 262-281.
Constant D., Kiesler S. and Sproull L. (1994). What's mine is ours, or is it? A study of attitudes about
information sharing. Information Systems Research, 5 (4), 400-421.
Doherty N.F., Coombs C.R. and Loan-Clarke J. (2006). A re-conceptualization of the interpretive
flexibility of information technologies: Redressing the balance between the social and the
technical. European Journal of Information Systems, 15 (6), 569-582.
Eisenhardt K.M. (1989). Building theories from case study research. Academy of Management
Review, 14 (4), 532-550.
Haythornthwaite C. (2006). Articulating divides in distributed knowledge practice. Information,
Communication &Society, 9 (6), 761-780.
Henderson J.C. and Venkatraman N. (1993). Strategic alignment: Leveraging information technology
for transforming organizations. IBM Systems Journal, 32 (1), 4-16.
Hevner A.R., March S.T., Park J. and Ram S. (2004). Design science in information systems research.
MIS Quarterly, 28 (1), 75-105.
Hargadon A. and Sutton R.I. (1997). Technology brokering and innovation in a product development
firm. Administrative Science Quarterly, 42 (4), 716-749.
Iivari J. (2007). A paradigmatic analysis of Information Systems as a design science, forthcoming in
Scandinavian Journal of Information Systems. Draft 27p, ask the newest version from the author
juhani.iivari@oulu.fi .
Järvinen P. (2004). On research methods, Opinpajan kirja, Tampere, Finland.
Levitin A.V. and Redman T.C. (1998). Data as resource: Properties, implications, and prescriptions.
Sloan Management Review, 40 (1), 89-101.
March S.T. and Smith G.F. (1995). Design and natural science research on information technology.
Decision Support Systems, 15 (4), 251-266.
Markus M. L., Majchrzak A. and Gasser L. (2002). A design theory for systems that support emergent
knowledge processes. MIS Quarterly, 26 (3), 179-212.
Parberry I. (1989). A guide for new referees in theoretical computer science. SIGACT News, 20 (4),
92-109.
Reeves C.A. and Bednar D.A. (1994). Defining quality: Alternatives and implications. Academy of
Management Review, 19 (3), 419-445.
Sahay S. and Robey D. (1996). Organizational context, social interpretation, and the implementation
and consequences of geographic information systems. Accounting, Management and Information
Technology, 6 (4), 255-282.
1396

Smith A. J. (1990). The task of the referee. Computer, 23 (4), 65-71.
Van Aken J.E. (2004). Management research based on the paradigm of the design sciences: The quest
for field-tested and grounded technological rules. Journal of Management Studies, 41 (2), 219-246.
Van Aken J.E. (2006), private communication.
Van der Heijden H. (2004). User acceptance of hedonic information systems. MIS Quarterly, 28(4),
695-704.
1397