What is Research Rigor? Lessons for a Transdiscipline

Abstract

Aim/Purpose: Use of the term “rigor” is ubiquitous in the research community. But do we actually know what it means, and how it applies to transdisciplinary research? Background: Too often, rigor is presumed to mean following an established research protocol scrupulously. Unfortunately, that frequently leads to research with little or no impact. Methodology: We identify a sample of 62 articles with “rigor” in the title and analyze their content in order to capture the range of perspectives on rigor. We then analyze how these findings might apply to informing science. Contribution: This paper offers an approach to defining rigor that is theory based and appropriate for transdisciplinary research. Findings: Rigor definitions tend to fall into one of two categories: criteria-based and compliance-based. Which is appropriate depends on the research context. Even more variation was found with respect to relevance, which is often used as a catch-all for research characteristics that aren’t associated with rigor. Recommendations for Practitioners: Recognize that when researchers are referring to rigor and relevance, they of-ten mean these to apply to other researchers rather than to practice. When funding research, it is important to understand who the rigor and relevance are directed towards. Recommendations for Researchers: When using the term “rigor”, think carefully about which meaning is intended and be transparent about that meaning in your writing. Impact on Society: A great deal of public money is invested in achieving research rigor. Society should be aware of what it is buying with that funding. Future Research: Developing a better understanding of research fitness and the factors that contribute to it.

Full text

Volume 23, 2020
Accepting Editor Eli Cohen │Received: January 28 2020│ Revised: March 25, 2020 │ Accepted: March 26,
2020.
Cite as: Gill, T. G., & Gill, T. R. (2020). What is research rigor? Lessons for a transdiscipline. Informing Science:
The International Journal of an Emerging Transdiscipline, 23, 47-76. https://doi.org/10.28945/4528
(CC BY-NC 4.0) This article is licensed to you under a Creative Commons Attribution-NonCommercial 4.0 International
License. When you copy and redistribute this paper in full or in part, you need to provide proper attribution to it to ensure
that others can later locate this work (and to ensure that others do not accuse you of plagiarism). You may (and we encour-
age you to) adapt, remix, transform, and build upon the material for any non-commercial purposes. This license does not
permit you to use this material for commercial purposes.
WHAT IS RESEARCH RIGOR?
LESSONS FOR A TRANSDISCIPLINE
T. Grandon Gill*
University of South Florida, Tampa,
FL, USA
grandon@usf.edu
Thomas R. Gill
University of South Florida, Tampa,
FL, USA
thomasgill@usf.edu
* Corresponding author
ABSTRACT
Aim/Purpose
Use of the term “rigor” is ubiquitous in the research community. But do we
actually know what it means, and how it applies to transdisciplinary research?
Background
Too often, rigor is presumed to mean following an established research proto-
col scrupulously. Unfortunately, that frequently leads to research with little or
no impact.
Methodology
We identify a sample of 62 articles with “rigor” in the title and analyze their
content in order to capture the range of perspectives on rigor. We then analyze
how these findings might apply to informing science.
Contribution
This paper offers an approach to defining rigor that is theory based and appro-
priate for transdisciplinary research.
Findings
Rigor definitions tend to fall into one of two categories: criteria-based and
compliance-based. Which is appropriate depends on the research context. Even
more variation was found with respect to relevance, which is often used as a
catch-all for research characteristics that aren’t associated with rigor.
Recommendations
for Practitioners
Recognize that when researchers are referring to rigor and relevance, they often
mean these to apply to other researchers rather than to practice. When funding
research, it is important to understand who the rigor and relevance are directed
towards.
Recommendations
for Researchers
When using the term “rigor”, think carefully about which meaning is intended
and be transparent about that meaning in your writing.
Impact on Society
A great deal of public money is invested in achieving research rigor. Society
should be aware of what it is buying with that funding.

What is Research Rigor?
48
Future Research
Developing a better understanding of research fitness and the factors that con-
tribute to it.
Keywords
rigor, relevance, resonance, interdisciplinary, transdisciplinary, research
INTRODUCTION
One of the most insulting things you can say to a group of academics is that their research isn’t rig-
orous. This is hardly surprising given the importance ascribed to rigor in our doctoral training and in
the subsequent review and publication process. What is more surprising is that there is no clear con-
sensus regarding what “rigor” means. Does the progress of our careers truly depend on a concept
that is so fuzzy?
Understanding the precise nature of rigor is particularly challenging in transdisciplinary research, such
as that emphasized by informing science. To the extent that any consensus on the definition of rigor
exists, that consensus is most likely to exist within a discipline or subdiscipline. Given the large dif-
ferences between how research is conducted in diverse fields—for example, design science research
contrasted with finance research—we would not expect that a universal recipe for rigor is likely serve
the needs of all research methods. Nevertheless, as a matter of definition, transdisciplinary research
necessarily draws upon findings and approaches from multiple disciplines. How can the rigor of the
resulting research product be assured? Especially if we aren’t sure what rigor is.
In the present paper, we survey how research rigor is defined and assessed across a broad range of
disciplines, drawn principally from the social sciences. Our goal is to develop a better understanding
of the core aspects of rigor that can be used across disciplines, particularly in the context of interdis-
ciplinary or transdisciplinary research. Our hope is that this analysis may help researchers better as-
sess the role that rigor should play in their research.
Because usage of the term rigor in research is extremely widespread, we begin by describing the pro-
cess through which 62 social science research articles dealing specifically with rigor were assembled.
We then identify the subset of 11 papers where a definition for rigor was included and summarize the
differences between definitions. We also attempt to discern different definitions of rigor from the
context in which the term is used across the entire set of 62 papers. In doing so, we find considerable
variation in where rigor is assumed to exist across the different stages of the research process (e.g.,
research design, execution, analysis).
The analysis of these articles produced several rigor-related issues that are then discussed. These in-
clude:
• What are the research outcomes that achieving high levels of rigor are expected to produce? Research rigor
does not come for free. Its costs can include economic costs, time costs, and can also extend
to accepting constraints on methodologies and on the selection of appropriate topics. Given
the various definitions of rigor, what benefits are expected to compensate for these costs?
• What is the relationship between rigor and relevance? In nearly 60% of the papers examined, the re-
lationship between rigor and relevance was considered. Interestingly, very different relation-
ships were often posited.
• How can existing informing science research contribute to our understanding of rigor? We relate rigor (and
relevance) to certain concepts—such as diffusion, filters, and fitness—that have been exam-
ined in informing science research.
We conclude the paper by considering how rigor might be defined in a manner that supports trans-
disciplinary research in informing science.

Gill & Gill
49
METHOD
The term “rigor” is widely used in the research literature. For example, a simple Google Scholar
search consisting of rigor OR rigour produced over one million records; for purpose of comparison, a
similar search consisting solely of the term research produced just over eight million records. Because
of this wide usage, an exhaustive study of how rigor is defined and used was not possible. Instead,
the following protocol was used to identify a representative body of literature that could be exam-
ined:
1. We constrained the search to articles with rigor (or rigour) in the title. The rationale here was that pa-
pers that focused specifically on rigor seemed would be the best place to find examples of
how it was defined and used. This reduced the number of records to 8220.
2. We limited our search to articles with a substantial number of citations. This appeared to be the best
way to focus one widely held perspectives on rigor. We were concerned, however, that this
approach would unnecessarily bias our search against recent articles (i.e., articles that had not
been around long enough to accumulate many references. For this reason, we adjusted our
cut points for inclusion as follows:
a. Articles published before 2015: at least 100 citations according to Google Scholar
b. Articles published 2015-2017: at least 50 citations
c. Articles published 2018-2019: all articles with promising titles were included
3. We chose to focus on articles in the social sciences only. Our assumption was that research in the natu-
ral sciences, life sciences, and humanities was likely to be conducted in ways sufficiently dif-
ferent from most social science research and that including them would unrealistically ex-
pand the range of definitions that we would find. The criteria for elimination was based on
the article title and publication outlet. The main impact here was the elimination of a surpris-
ing large body of literature from the field of nursing.
4. We included only articles relating to rigor applied to research. As we conducted the search, it became
clear that rigor could be applied in very different contexts. For example, it can refer to spe-
cific medical conditions (e.g., “rigor mortis”) and can also be applied to teaching (e.g., “a rig-
orous program of study”). While the latter usage certainly bears some relationship to the us-
age in the context of research, we considered it outside of the scope of the current paper.
Most of the selection process was accomplished through the article title and outlet. Five pa-
pers were subsequently eliminated based on content.
All articles meeting the four criteria for inclusion were downloaded, a total of 62 articles. For imaged
documents, the OCR capability incorporated into Adobe’s Acrobat™ product was used to create
searchable text.
Each author then independently searched for all occurrences of the term rigor (including rigour, rig-
orous, and rigourous) within each document, looking for text in which the term was defined or
where it was used in a manner that shed light on its intended meaning. In each case, the relevant text
was copied into each author’s own spreadsheet.
After a first pass, in which the two authors had an 81% level of agreement of whether rigor was de-
fined in each article, we collaborated to reach a 100% consensus. Interestingly, the quotes we both
collected independently were highly consistent. The initial disagreements largely arose from interpret-
ing the text. The challenges included the following (bold emphasis added by the authors):

What is Research Rigor?
50
• Use of conjunctions making it difficult to disambiguate rigor from other constructs. For exam-
ple:
Although effective writing and transparency are necessary, the
rigor and quality
of inductive papers rest
on three fundamental criteria, as follows. First, and as with all strong theory, is the emergent theory internally
coherent and parsimonious? ... Second, are the constructs or themes convincingly grounded in compelling data?
... Third, does the research provide rich and unexpected insights? (Eisenhardt et al, 2016, pp. 1120-
1121)
• Presence of multiple definitions that are not necessarily consistent with each other. For example:
Rigor is described as demonstrating integrity and competence
within a study. ... Schutz’s
first postulate of logical consistency is similar to the description by Horsfall, Byrne-Armstrong, and Higgs …
of
rigor in qualitative research, which involves in-depth planning, careful attention to
the phenomenon under study, and productive, useful results
. Descriptions of theoretical
ri-
gor involve sound reasoning and argument and a choice of methods appropriate to
the research problem
. (Fereday & Muir-Cochrane, 2006, p. 82)
• Ambiguity as to whether a statement defines rigor or identifies characteristics that are likely to
accompany, or arise as a consequence of, rigor. For example:
We argue that
rigorous PD [participatory design] work exhibits qualities that are coher-
ent
, e.g., it is based on an epistemology that accommodates the values that drive the effort, involves stakehold-
ers in ways that reflect these foundations and accordingly defines and delivers its meaningful outcomes. (Frau-
enberger et al., 2015, p. 103)
Where rigor could not be disambiguated from other concepts, we treated the article as not defining
rigor. On the other hand, where multiple definitions were proposed, we classified that article as de-
fining rigor. This was consistent with our expectation that multiple definitions would be expected in
transdisciplinary research. Finally, we discussed each ambiguous usage of rigor to determine if a defi-
nition could reasonably be assumed from the context, coming to a consensus.
After identifying definitions of rigor and contexts where rigor’s meaning was suggested by usage, we
sought to classify different perceptions of rigor. After considerable discussion and analysis of the
text, we chose a scheme based upon where (in the research process) rigor was emphasized. The pro-
cess model we used is presented in Figure 1.
Figure 1: Stages of the research process
The stages were defined as follows:
1. Design: Rigor was specifically mentioned as being critical in the research design or in the
choice of an appropriate research methodology. For example:
This has motivated some postpositivist researchers to
carefully design their studies
, using
quantitative methods to minimize “bias” or “subjectivity.” Over time, these efforts have become
standardized as criteria
to ensure the rigor of the work
. (Barusch et al., 2011, p. 13)
2. Execution: The importance of systematically and faithfully executing the chosen research
methodology was indicated to be a key element of rigor. For example:

Gill & Gill
51
the current trend in
methodological rigor
is to focus on rigor per se with the concomitant op-
erationalization of concepts and
systematic utilization of procedures that foster
standardization
. (Armour et al., 2009, p. 102)
3. Analysis: Quality of reasoning, application of logic or the use of appropriate statistical tech-
niques was specifically mentioned as being critical to rigor. While we had originally consid-
ered this a subset of execution, certain methodologies—such as those involved with the ri-
gor of theory-building—seemed better served with a separate category. For example:
In conclusion,
rigor in research is the strength of the chain of reasoning
(Biggs &
Büchler, 2007, p. 69)
4. Product: Rigor was specifically tied to the final product of the research, such as an article. We
selected this category when articles referred to the rigor of the description or indicated that
transparency of the research was a critical component rigor. For example:
Similarly,
rig or
—often seen as one of the key weaknesses of case studies—often seems to lie in the
eye of the beholder and
may even involve ‘‘persuading’’ readers and reviewers
… of
the ‘‘credibility’’ of methodological procedures (Gibbert & Ruigrok, 2010, p. 710)
5. Outcome: Some articles suggested that rigor could only be achieved where a particular out-
come of the research—beyond publication—is achieved. For example:
Interpretive
rigor goes beyond the sound application of method to focus on
moral and aesthetic dimensions of scholarly quality
. It pushes researchers to engage
strategies that
directly contribute to worthwhile human purposes
, not only from their
own perspective, but from the perspectives of other social actors involved in the research (Dodge et
al., 2005, p. 297)
Both authors classified all 62 articles independently, with nearly all the articles appearing in multiple
research stage categories. The classifications were then compared. While the initial level of agreement
was disappointingly low (70%), a discussion regarding when the design category was applicable elim-
inated about half the differences in classification. At that point, we went through each article and
arrived at a consensus on the final classifications.
Later, during the analysis process, it became clear that one of the most interesting outcomes of the
process was the high level of variation in how the relationship between rigor and relevance, a topic
explored in 60% of the articles studied. At this point, the original process of gathering quotes and
definitions was repeated for both the terms relevance and relevant. These results are referred to in the
discussion section of the current paper.
RESULTS
In this section we report the results of applying the preceding methodology to the collection of arti-
cles. We begin by looking at the definitions, then examine the distribution of rigor considerations
across the different stages of research.
DEFINITIONS
Of the 62 articles examined, 24% (15) contained statements meeting our criteria for being classified
as definitions. Of the 15 articles, only 6 appeared to have chosen a single definition (i.e., Armour et
al., 2009; Gulati, 2007; Hasson & Keeney, 2011; McAlister, 2016; Nunamaker et al., 2015; Ogawa &
Malen, 1991). The list of references and associated quotes are presented in tabular form in the Ap-
pendix.
Figure 2 presents a word cloud and frequency table extracted from the Appendix definition text,
constructed using words that appear more than once and removing words not related to the defini-
tion (e.g., research, rigor, relevance, etc.).

What is Research Rigor?
52
Figure 2: Frequency table and word cloud for selected terms in Appendix definitions.
RIGOR B Y RESEARCH STAGE
When the articles were classified according to the research stage where rigor was described, more
than two thirds (42 out of 62) listed multiple categories. The distribution of research stages within the
rigor articles is presented in Figure 3.
Figure 3: Counts of articles (out of 62) by the research stage where rigor was described
In Table 1, key themes relating to how rigor is considered are summarized, organized by research
stage.

Gill & Gill
53
Table 1: Key themes of rigor articles, organized by research stage
Cite
Design
Execution
Analysis
Product
Outcome
Comments
McAlister (2016) X
This paper emphasizes proper method selection as the key to
rigor, arguing that doctoral programs should have an up to date
methods seminar to help future researchers choose the method
most conducive to rigor in each specific research project.
Barusch (2011), Combs
(2010), Daft & Lewin
(2008), Eisenhardt
(1991), Erickson &
Gutierrez (2002)
X
The Barusch article studies the strategies used by researchers to
promote rigor in their work. Combs’ article serves as a warning
that researchers not let increased capabilities with regard to sam-
ple size lead them to neglect rigor and relevance. Daft’s paper
focuses more on relevance than rigor and asserts that organiza-
tional studies research should value academic relevance over
practical relevance. Eisenhardt’s article compares the quality of
theory derived from single and multiple case research partly
through the lens of rigor. Erickson & Guitierrez call for more
broad standards for scientific rigor in the realm of educational
research.
Kieser & Nicolai
(2005), Kincheloe
(2011), Lu & Shulman
(2008), O’Dwyer &
Unerman (2016),
Onwuegbuzie et al.
(2009), Ross et al.
(2010), Sculley et al.
(2018)
X
Kieser & Nicolai’s paper focuses on the rigour-relevance gap,
particularly the communication processes between scientists and
practitioners. Kincheloe argues that bricolage (interdisciplinarity)
will naturally lead to greater rigor. Lu & Shulman profess that
computerized qualitative data analysis is a great way to achieve
higher rigor. Odwyer argues that there is great potential for pa-
pers that are both rigorous and relevant in the field of account-
ing, and that it is just those papers that will end up leading to
meaningful changes. Onwueguzie’s paper analyzes the rigor of
various methods of focus group research, concluding transcript-
based analysis to be the most rigorous. Ross came to the conclu-
sion that research must be rigorous in order for it to have rele-
vance in the field of education technology. Sculley argues that
technological advances must be matched by advances in rigor in
machine learning research.
Aguinis et al. (2010),
Davenport et al. (1999),
Kieser & Leiner (2009)
X
These papers share a commonality in their primary focus, viz.,
the rigor-relevance gap. Keiser and Leiner take a pessimistic
view, asserting that the gap is unbridgeable as the relationship
between researchers and practitioners currently stands. The other
two papers are more optimistic. Davenport et al. believe that
greater relevance could be achieved by emulating practitioner
focused consulting research, all the while improving upon it with
regard to rigor. Aguinis calls for a more customer centric ap-
proach to reporting with greater focus on statistical significance,
effect size, and practical significance.
Gutiérrez & Penuel
(2014), Hodgkinson &
Rousseau (2009), Lee &
Hubona (2009)
X
The Gutierrez paper argues that relevance should be made a
criterion of rigor. Hodgkinson and Rousseau argue that both
rigor and relevance are important, but there is a limit to how
much they can both be present in a single study. The Lee paper
stresses not relevance but summative validity.

What is Research Rigor?
54
Cite
Design
Execution
Analysis
Product
Outcome
Comments
Applegate & King
(1999), Snow (2015) X
X
While Snow believes that rigor is important, her article focuses
more on putting what has been learned in education research
into practice. Applegate and King present a case regarding an
assistant professor who had submitted a paper which was praised
for rigor but criticized for providing few new insights. These
papers are different in that Snow’s focuses more on finding a
way to get practitioners to adopt new research finding whereas
Applegate & King’s highlights the struggle of one researcher to
keep up with hot topics in research before they become oversat-
urated
Rajagopalan (2019) X
X
Argues that rigor and relevance are not mutually exclusive and
that both are necessary to have a lasting impact. Believes that the
availability of more sophisticated methods can be a double-edged
sword. On the one hand these methods can potentially increase
rigor, on the other hand this does not matter if they are only
used to delve into the more mundane aspects of research. The
core question must always be the main focus
Armour et al. (2009),
Collier et al. (2012),
Robey & Markus
(1998), Rosas & Kane
(2012)
X
X
The Collier, Armour, and Rosas papers focus on the rigor of
concepts (formation, mapping, and operationalization), whereas
Robey focuses on ways to close the rigor-relevance gap.
Bandara et al. (2015),
Grover et al. (1993) X
X
Grover focuses on rigor in survey research while Bandera focus-
es on rigor in literature reviews. They both stress the importance
of both methodology and reporting in their respective research
strategies.
Dodge et al. (2005),
Hasson & Keeney
(2011)
X
X
Dodge argues that researchers should not assume that rigor au-
tomatically begets relevance, as it can cause them to become
complacent in the search for findings that are useful and action-
able. Hasson & Keeney focus more on rigor than on relevance,
highlighting the dependability rather than the utility of research
outcomes
Eisenhardt et al. (2016),
Gasson (2004), Web-
ster (1978)
X
X
Webster’s article attempts to help researchers reap the benefits
of new models and the potential for greater rigor in industrial
marketing research. Gasson’s article focuses on qualitative re-
search and its reputation for low rigor, arguing that there need be
a different set of rules for rigor in qualitative research. Eisen-
hardt likewise suggests that the typical standards for rigor are ill
suited to certain problems, viz., Grand Challenges: “highly signif-
icant yet potentially solvable problems such as urban poverty,
insect-borne disease, and global hunger” (p. 1113), for which
inductive methods could prove particularly useful.
Frank & Landström
(2016) X
X
Argues that entrepreneurship research leans more towards rigor
than relevance because rigor is more in their comfort zone. They
are comfortable assessing rigor whereas they are not comfortable
assessing relevance from a practitioner standpoint. Their pro-
posed solution entails creating more of a dialogue between the
researchers and the stakeholders

Gill & Gill
55
Cite
Design
Execution
Analysis
Product
Outcome
Comments
Barbour (2001), Biggs
& Büchler (2007),
Darke et al. (1998),
Dubé & Paré (2003),
Gulati (2007),
Nunamaker et al.
(2015), Varadarajan
(2003), White (2002),
X
X
X
Barbour suggests some fixes (e.g., grounded theory, multiple
coding) to make qualitative research papers more rigorous but
concludes by asserting that true rigor can only be achieved
through a general knowledge of data analysis and qualitative re-
search design. Darke lays out some suggestions for making case
study research more rigorous such as choosing the right data
analysis technique. Dube & Pare’s article also focuses on case
research, laying out attributes for rigor in the areas of research
design (e.g., clear research questions, theory of interest,) data
collection (e.g., multiple methods, triangulation,) and data analy-
sis (e.g., logical chain of evidence, empirical testing). Varadarajan
calls for more consensus with regard to the definitions of con-
structs in marketing research as an aide to both rigor and rele-
vance. White suggests ways to increase both rigor and relevance
in Asian management research by aspiring to achieve a general-
izable understand of underlying phenomena. Biggs & Buchler
argue that regardless of the type of research, rigor always comes
down to a strong chain of reasoning. Gulati argues that the belief
that one has to choose between rigor and relevance is damaging
to research overall, and gives recommendations for achieving
both such as a greater focus on theory building and paying great-
er attention to managerial sensibilities. Nunamaker makes a simi-
lar argument, referring to researchers who vie to achieve both
rigor and relevance as Last Mile Researchers.
Gioia et al. (2013) X
X
X
Makes a case for the potential for rigor in qualitative research,
arguing that one of the reasons that qualitative research has a
reputation for low rigor is that researchers limit their conception
of rigor to the extension of old concepts, even when new ones
are called for.
Ivarsson & Gorschek
(2011), Ogawa &
Malen (1991), Poland
(1995), Rand & Rust
(2011), Stewart et al.
(2017), Tushman et al.
(2007), Vermeulen
(2005), Walt (1999)
X
X
X
Poland focuses on transcription quality as means to greater rigor
in research involving interviews. Rand & Rust suggest agent
based models must be verified and validated before they can be
considered rigorous. Stewart focuses on rigor in qualitative re-
search, concluding that credibility and trustworthiness are essen-
tial to achieving it. Tushman suggests that it is possible to have
both rigor and relevance and that aspiring to have both is what
separates business schools from other academic institutions.
Vermeulen also argues that rigor and relevance are not mutually
exclusive, but posits that they are rarely seen together because
the people conducting rigorous research are not asking questions
that are of interest to practitioners. Walt argues that greater rigor
can be achieved in political science research by focusing more on
testing theories and making them clearer and thus easier to find
flaws with. The Ivarsson and Gorschek article focuses on report-
ing, i.e., giving readers enough details to assess the rigor of the
study for themselves. This, they assert, is also necessary to
achieving relevance. Ogawa and Malen attempt to layout guide-
lines for achieving rigor in literature reviews including clear defi-
nitions of constructs and the search for contrary findings.

What is Research Rigor?
56
Cite
Design
Execution
Analysis
Product
Outcome
Comments
Sovacool et al. (2018) X
X
X
X
Sovacool vies to help other researcher achieve both rigor and
relevance in research, giving guidelines for both. He argues that
researchers should have a broad awareness of the various re-
search methods as this will both help them achieve rigor, and
help them be more humble and open minded, viz., in a better
state to seek out novel (relevant) ideas.
Frauenberger et al.
(2015) X
X
X
X
This paper focuses on achieving rigor in participatory design
research. By way of aiding researchers on the path to rigor, it
proposes four lenses through which to analyze a project: episte-
mology, values, outcomes, and stakeholders. The paper also as-
serts that rigor is not one size fits all, and that there may be pro-
jects that bear little resemblance to each other method wise, but
are nonetheless each rigorous in their own way
Fereday & Muir-
Cochrane (2006),
Nowell & Albrecht
(2018), Shrivastava
(1987)
X
X
X
X
X
The Fereday and Nowell articles both focus on rigor in qualita-
tive research, both likewise calling for a balance between induc-
tive and deductive methods. Shrivastava analyzes the presence of
both rigor and usefulness in strategic management research,
highlighting articles where both are present and suggesting that
the field in general has great potential to make simultaneous
strides in both.
DISCUSSION
Based on the articles surveyed, there appear to be two principal motivations for analyzing rigor:
1) To consider what constitutes rigor in non-positivist research approaches, e.g., phenomenological research
(e.g., Armour et al., 2009), qualitative research (e.g., Bandara et al., 2015; Biggs & Büchler,
2007; Lu & Shulman, 2008; Stewart et al., 2017) including grounded theory (e.g., Gasson,
2004), case study research (Darke et al., 1998; Eisenhardt, 1991; Ogawa & Malen, 1991;
Seuring, 1998), design research (Frauenberger et al., 2015), interview research (e.g., Hasson
& Keeney, 2011; Onwuegbuzie et al., 2009; Poland, 1995), bricolage (Kinchloe, 2011), action
research (e.g., Melrose, 2001), conceptual research (Collier et al., 2012; Rosas & Kane, 2012)
and modelling-based research (e.g., Rand & Rust, 2011).
2) To explore the relationship between rigor and relevance. Of the 62 articles studied, 47 (76%) referred
to relevance and considered its relationship to rigor. Indeed, separating out the elements of
research quality that were attributable to rigor vs. relevance proved to be a topic closely re-
lated to how rigor was defined.
In this section, we will examine these two topics. We will begin by looking at the challenges of defin-
ing rigor across a broad range of research methodologies and disciplines. We then consider the spe-
cifics of the rigor-relevance relationship—a relationship whose proposed nature varies considerably
across the articles surveyed.
ALTERNATIVE PERSPECTIVES ON RIGOR
As noted in the results section, there was considerable variation in views on the nature of rigor. Of
particular interest were two areas: 1) approaches to rigor and the 2) perceived impact of rigor.

Gill & Gill
57
Approaches to rigor
Most of the perspectives on rigor that we examined fell into one of two categories: compliance-based
or criteria-based. We now consider the similarities and differences between the two.
Compliance-based perspectives
. Many of the definitions and usages of rigor that we examined
were based strictly upon degree to which the research method complied with an accepted methodol-
ogy. Specifically, they focused on rigor as it related to (a) selection of an appropriate research meth-
odology, (b) fidelity in executing the methodology (and associated analysis), and (c) transparency in
documenting the use of the methodology. Of these three items, (b) was viewed as particularly critical,
as indicated earlier by the large number of articles falling in either or both of the execution and analy-
sis categories. An example of a definition fitting this category is:
Rigor, therefore, can refer either or both to methodological thoroughness and precision or
criteria used to judge the trustworthiness of the results. Methodological stringency and accu-
racy of the results are related because solidity in methods provides greater assurance that the
findings are valid. (Armour et al., 2009, p. 102)
Naturally, the nature of rigor varies according to the research methodology. For positivist, empirical
research using a large sample it would likely involve issues such as the choice of sampling approach,
the appropriateness of the statistical tests employed, whether the interpretation of the findings is
consistent with the results, and so forth. For qualitative, interpretive research, rigor might involve
considerations such selection of interviewees, approach used to capture and code interviews, use of
triangulation to confirm findings, techniques employed to reduce researcher preconceptions and bias,
and so forth.
The advance of compliance-based perspectives on rigor is that they can build upon previous applica-
tions of the same research method. Thus, over time, an implicit “rule book” for each method can
evolve. These rules can be taught to researchers unfamiliar with the method, can guide researchers
employing the method, and can provide standards that reviewers can follow.
The potential drawback of these approaches is that they focus nearly all attention on the method em-
ployed without stepping back and asking if the method is likely to produce useful results. For exam-
ple, 16 of the articles examined specifically mention replication in conjunction with rigor. On the
surface this makes sense; the ability to replicate a study in order to confirm its results is considered
critical in the natural sciences. In the context of the social sciences, however, the preponderance of
evidence suggests that expecting results to replicate across similar studies is optimistic. For example,
attempts to replicate research across different business disciplines have yielded disappointing results
(Gill, 2016b, pp. 134-135). Even attempts to replicate “textbook” psychological research conducted
under far more controlled conditions have failed to live up to expectations (Open Science Collabora-
tion, 2015). Thus, compliance-based rigor allows researchers to design and conduct research that cor-
responds as closely as possible to a previous study. If we have no reason to expect the results should
be the same or similar, however, why bother?
Criteria-based perspectives.
The issues associated with pure compliance-based approaches to ri-
gor—particularly as they apply to replication—have been recognized in some of the articles we ex-
amined. For example:
Two criteria appropriate for deductive research but NOT appropriate for inductive inquiry
include:
1) Is there evidence that the causal factors, processes, nature, meaning, and/or significance
of the phenomenon generalize to the broader population?
2) Are the findings able to be replicated in the sense that two researchers asking the same
question would come to the same interpretation of the data?

What is Research Rigor?
58
These two criteria, held sacred as cornerstones of rigor in deductive inquiry, seem to cause
the greatest amount of heartburn within the field of public management and its relationship
to inductive qualitative inquiry. If it is not generalizable and it does not replicate, how is that
possibly science? This results in on-going frustration among qualitative scholars as they at-
tempt to respond to criticisms of their design by reviewers, colleagues, and advisors in terms
of the lack of representative sampling and/or inter-rater reliability measures. (Nowell & Al-
brecht, 2018, p. 353)
To address this issue, some authors argue that research rigor should be assessed based on more glob-
al attributes of the research, what we are referring to as criteria-based rigor. In their analysis of rigor
in qualitative research, Nowell and Albrecht (2018) continue:
If we cannot assess inductive studies in terms of generalizability and replication, what are
valid criteria upon which they might be evaluated? In very global terms, rigorous inductive
research in public management can be judged on two core criteria:
1) Does the research design and its execution generate new insight into the causal factors,
processes, nature, meaning, and/or significance of a phenomenon of interest to the field? …
and
2) Is the account of these causal factors, processes, nature, meaning, and/or significance
within these cases trustworthy? … The trustworthiness and depth of insight of an inductive
study is manifest in its research design, execution, reporting. (p. 354)
The advantage of criteria-based assessments of rigor is that they can be readily adapted to very differ-
ent research approaches. For example, although the interpretation of a property such as “trustwor-
thiness” is likely to be very different for alternative methodologies, the concept itself is fairly univer-
sal. The obvious drawback to criteria-based approaches is along the same lines. Since they require
interpretation for each research project considered, they will necessarily be somewhat subjective. As a
consequence, the assessment of rigor for a given project will likely produce considerable disagree-
ment across different reviewers as readers.
Impact of rigor
Just as the articles studied illustrate differences in how rigor is assessed, they also differ in the ex-
pected impact of ensuring rigor in research. Some of the perceived impacts include the following:
• Credibility. Attention paid to rigor is seen to increase the credibility of research. Many of the char-
acteristics of rigorous research, such as being thorough, careful, systematic, and logical contribute
directly to this outcome. In business, this credibility is seen as one of the key factors that distin-
guishes academic research from other research sources, such as consulting. For example:
Readers are likely to be attracted to academic research reports because of this perceived neu-
trality, knowing that our results are more trustworthy than vendors’ claims and promotional
materials. (Davenport et al., 1999, p. 14)
• Consistency. Particularly where compliance-based rigor is enforced, within a given research context
different researchers should come up with similar findings. This quality is also referred to as relia-
bility and can be applied across a wide range of contexts (e.g., reliability of instruments, reliability
of different investigator ratings, statistical reliability tests).
• Replicability. The ability to reproduce the research in a different context, also particularly relevant
to compliance-based rigor. As noted previously, however, within the social sciences it is not clear
that we should expect results to replicate. Nevertheless, we should be able to replicate the meth-
od applied. This quality is closely related to transparency—describing to the degree that the ra-
tionale for all research decisions is made clear to the reader—is specifically mentioned in 16 of
the articles. For example:

Gill & Gill
59
rigour and transparency in the process is a major contribution to knowledge in Design Re-
search (Frauenberger et al., 2015, p. 99)
• Validity. Two thirds of the articles surveyed specifically related research rigor to its validity. Inter-
nal validity refers to the logical consistency of relationships described by the research. Construct
validity refers to the degree to which concepts or variables used or developed by the research can
be supported in terms of both their independent contribution to understanding the phenomenon
being studied and the degree to which they can be distinguished from other constructs. External
validity refers to the degree to which research results are expected to apply outside of the research
context and is closely related to generalizability.
• Quality. In many of the articles studied, rigor and research quality are closely associated. One in-
terpretation is that rigor is the source of quality, although it might also be argued that rigor and
quality are deemed to be synonymous. Since quality is not defined, it is difficult to discern which
interpretation applies. Some examples illustrate the challenge:
Although effective writing and transparency are necessary, the rigor and quality of inductive
papers rest on three fundamental criteria (Eisenhardt, et al., 2016, p. 1120).
As the main indicator of academic quality rigour can be more easily evaluated because re-
searchers are trained in scientific methods and the identification of research gaps (Frank &
Landström, 2016, p. 66).
For ensuring rigor, quality criteria have been put forward which should be obeyed. (Seuring,
2008, p. 128).
In being asked for this paper to propose directions for improving the relevance (meaning-
fulness and utility) and quality (rigor and credibility) of research, we accept that many alter-
native viewpoints, each having its own compelling rationale, are likely to exist. (Ross et al.,
2010, p. 30)
A significant issue raised by some of the perceived impacts of rigor involves how they impact the
usefulness of the construct. Where rigor’s impact depends on the nature of the research product (e.g.,
transparency) or on the research outcome (e.g., quality), it will be hard to ensure rigor prior to know-
ing the results of the research. This presents a significant barrier to the research designer. Other ex-
pected impacts, such as credibility and validity, can be planned for through careful attention to de-
sign, executive and analysis.
THE RELATIONSHIP BETWEEN RIGOR AND RELEVANCE
As previously noted, more than three quarters of the articles surveyed addressed the question of the
tradeoff between rigor and relevance. Understanding this tradeoff is important in clarifying what is
meant by rigor, as relevance is frequently treated as those aspects of research that are not achieved
through rigor; hence differences in the expected impact of rigor will likely be accompanied by com-
plementary differences in the expected impact of relevance. Dodge et al. (2005, p. 287) voice the fol-
lowing complaint: “The traditional definition of ‘rigor,’ which assumes an automatic connection to
relevance, is particularly problematic.” Gulati (2007, p. 775) further notes that debates relating to ri-
gor vs. relevance often start from the assumption that the two qualities are antithetical.
Alternative perspectives of relevance
While we have noted that the term rigor is frequently used without definition, the situation is much
worse with regards to relevance. Indeed, of the 47 articles that addressed relevance, only 6 offered
anything remotely resembling a definition. These definitions varied widely, as shown in Table 2.

What is Research Rigor?
60
Table 2: Quotes that serve to define relevance
Definition
Source
Relevance refers to the potential of research (questions and findings) to enable practitioners
“to make informed choices about important practical problems and to implement solutions
to them effectively” (AOM 2004). It also refers to the extent to which research addresses the
challenges that practitioners face in their work and whether the questions and findings reso-
nate with practitioners’ experience, shedding new light on existing problems in ways that are
actionable. (p. 288)
Dodge et al.
(2005)
Per Webster’s dictionary, relevance is “relation to the matter at hand” or “practical and espe-
cially social applicability” (p. 775)
Gulati (2007)
In an applied research field such as software engineering, the transfer and widespread use of
research results in industry ultimately determine the relevance and success (p. 366)
Ivarsson &
Gorschek
(2011)
Relevance is the degree to which research contributes directly to improving outcomes of in-
terest to practitioners in the field, that is, solves an important class of problems. (p. 41)
Nunamaker et
al. (2015)
For educational technology research to help solve real-world educational problems, we advo-
cate that studies increasingly reflect two qualities. One is to achieve balance between rigor
(internal validity) and relevance (external validity) (p. 24). In being asked for this paper to
propose directions for improving the relevance (meaningfulness and utility) and quality (rigor
and credibility) of research, we accept that many alternative viewpoints, each having its own
compelling rationale, are likely to exist. (p. 30)
Ross et al.
(2010)
While conventional academic disciplines are typically about a quest for understanding (rigor)
with little thought of use (relevance), business schools, and professional schools more gener-
ally, are about both - operating in Pasteur’s Quadrant. …Consulting firms, unlike business
schools, are focused on meeting clients’ needs (relevance) but are less concerned with general
theory building or carefully controlled research (rigor). (p. 347)
Tushman et al.
(2007)
While all of these definitions suggest a relationship between relevance and practical usefulness, they
leave two important areas of ambiguity:
• Is the “usefulness” potential or must it be realized? In other words, can relevance be established prior to
the research outcomes being known or must we know the impact of research before we deem it
to be relevant?
• What community determines usefulness? Research often has two distinct audience, the community of
researchers within a discipline and the community of practitioners to which the research might
apply. Depending upon which community we are talking about, the meaning of relevance would
be very different. For example, relevant research to the business research community might in-
volve identifying and filling a gap in the literature; for the business practice community, it might
involve addressing an important question facing managers and providing actionable, practical
recommendations.
For our purposes, we believe that both rigor and relevance are best defined in a manner that offers
the researcher insights with respect to conceiving and designing research. Otherwise, they devolve
into terms for classifying research outcomes, in which case we would need to define new terms (e.g.,
potential for rigor, potential for relevance) to guide design. We are particularly attracted to a defini-
tion associating relevance with the type of research questions being asked, i.e., Are the questions
guiding the research of importance to the community of practice? To quote Vermeulen (2005):
In any study, it is the research question that was asked in the first place that determines the
usefulness of the study’s findings. Thus, academic answers often lack practical meaning be-
cause the questions that were asked to start with lacked relevance. Asking questions that are
of importance to reality, while not making concessions in terms of rigor in developing theory

Gill & Gill
61
and empirical evidence, would provide most value. Relevance is then found in the question,
rigor in the method applied to provide the answer. (p. 979)
The rigor-relevance relationship
While many of the articles studied suggest that rigor and relevance are treated as diametrically op-
posed in past research (e.g., Dodge et al. 2005, p. 287), we could not find any articles that actually
took that position. We did, however, find a wide range of proposals regarding how rigor and rele-
vance were related. We now consider some of these.
Prerequisite Relationships
: The first type of relationship we observed assumes that either rigor or
relevance is a prerequisite of the other. Both types of relationship have been proposed, as shown in
Figure 4.
Figure 4: Proposed prerequisite relationships between rigor and relevance

What is Research Rigor?
62
For example, Van Weele and Van Raaij (2014) assert the following:
When we speak about relevance, we do not view rigor and relevance as trade-offs, but we
prefer to view methodological rigor in service of research relevance. Management research
cannot be truly relevant, if it has not been executed rigorously. (p. 64)
On the other hand, the opposite relationship has been proposed. For example:
Making relevance to practice a key criterion of rigor is an important step toward more equi-
table and consequential research. Making relevance to practice a key criterion of rigor is an
important step toward more equitable and consequential research. (Gutiérrez & Penuel,
2014, p. 22)
Rig or and Relevance as Separate Dimensions
. If rigor and relevance are related through prereq-
uisite relationships, they are inseparable. Another proposed relationship between the two concepts is
as separate and independent relationships. One example of this involves a slight variation on the
graph used to visualize Pasteur’s quadrant, as shown in Figure 5. In this depiction, “pure research” is
treated as being highly rigorous, with little obvious immediate relevance. At the opposite extreme,
applied research is conceived as being highly relevant with little attention being paid to formal rigor.
The “ideal” research takes place in Pasteur’s Quadrant, with highly useful questions being tackled
with rigorous methods.
Figure 5: Pasteur’s Quadrant Chart, adapted from Tushman, et al. (2007, p. 347)
The obvious question the Figure 5 depictions raises is the following: If use-inspired, opportunities to
conduct rigorous research abound, why would we ever choose to operate in any other quadrant? This
question is particularly applicable in fields like management, where a rapidly changing, complex envi-
ronment make it unlikely that long standing principles will be found through a pure research process.
For this reason, another category of relationships focus on tradeoffs between rigor and relevance.

Gill & Gill
63
Rigor-Relevance Tradeoffs
. A couple of different approaches to characterizing the tradeoffs be-
tween rigor and relevance exist. Starting with the same rigor and relevance dimensions as Figure 5,
the left-hand side of Figure 6, adapted from Robey and Markus (1998, p. 9), presents a curve show-
ing how research can vary from academic-focused to practitioner-focused. They posit a middle
ground, referred to as “consumable academic research” that would maintain substantial rigor while
being presented in a form accessible to practitioners.
Figure 6: Research curve adapted from Robey & Markus (1998, p. 9)
and their re-interpretation as microeconomic indifference curves
The shape of the Robey-Markus curve closely resembles that of indifference curves, familiar to most
veterans of introductory microeconomics. Each curve, illustrated on the right-hand side of Figure 6,
represents rigor-relevance combinations of equal utility—the decision maker does not care (i.e., is
indifferent to) which rigor/relevance combination on a particular curve is selected.
Another view of a rigor-relevance tradeoff is proposed by Davenport et al. (1999), shown on the left-
hand side of Figure 7. They propose a curve that represents a threshold between impactful and non-
impactful research. From a “big picture” perspective, the curve proposes that impactful research can
arise from either very rigorous research or from very relevant research or from research that is a
combination of the two.
We are somewhat suspicious of the convex shape that the authors used to draw their curve. The im-
plication of that shape is that once you reach a very high level of rigor, relevance does not really mat-
ter; the same applies to relevance—at very high levels, rigor adds little or nothing to impact. On the
other hand, their impact curve bears a striking resemblance to another familiar microeconomic curve:
the production possibilities curve (shown on the right hand side of Figure 7). The production possi-
bilities curve identifies possible combinations of two goods that can be produced with a given pro-
duction capacity. Any combination on or within the curve is feasible to produce; any combination
outside the curve requires additional capacity. The curve itself therefore represents the best tradeoffs
between the two “goods” (in this case, rigor and relevance) that can be achieved.

What is Research Rigor?
64
Figure 7: Rigor-relevance impact curve (Davenport et al., 1999, p. 23)
and production possibilities curve
Again drawing on basic microeconomics, if we combine the utility curves and the production possi-
bilities curve we can identify the “optimal” combination of the two goods. The optimum point oc-
curs where the production curve is tangent to the utility curve; such a point will always exist where
the production curve is convex and the indifference curves are concave. This is illustrated in Figure 8
for rigor and relevance.
Figure 8: Utility-maximizing balance between rigor and relevance
We are quick to point out that Figure 8 does not imply that there is an “optimal” balance between
rigor and relevance, nor is such an optimum suggested by the literature. Each individual researcher or
research consumer will have their own indifference curves. The shapes of these curves can vary sub-
stantially. As illustrated in Figure 9, very different utility-maximizing combinations can result. A
strong preference for either rigor or relevance produces curve with a very sharp bend—becoming a
right angle in the limit—such that only increases in the preferred dimension produce significant im-
provements in utility.

Gill & Gill
65
Figure 9: Differing preferences (utility curves) can lead to very different
rigor-relevance combinations
Sufficiency of rigor and relevance
Another question raised by the articles surveyed relates to the degree to which achieving rigor and
relevance is sufficient to achieve research effectiveness—an intentionally vague term that we are us-
ing to include aspects of the research such as impact and quality. The earlier Figures 4 through 9 all
seem to imply that research effectiveness can be achieved through rigor and relevance. Some of the
articles, however, argue that other qualities are needed. For example, Sovacool et al. (2018) identify
rigor, novelty and style as the source of research quality, with relevance included in the four core el-
ements necessary to achieve quality, e.g.,
Although the later parts of this Review will explore how to improve aspects of novelty, rigor,
and style, a useful starting point is to consider four core elements: 1) asking concise, interest-
ing, socially relevant, and answerable research questions; 2) applying and testing theoretical
constructs or conceptual frameworks; 3) clearly stating research objectives and intended con-
tributions; and 4) developing an appropriate research design. (p. 13)
Rajagopalan (2019, p.1) refers to rigor, relevance and resilience, the latter being a measure of the ro-
bustness of research. Aguinis et al. (2010, p. 512) refer to rigor relevance and practical impact. Ba-
rusch et al. (2011) similarly state:
Careful attention to rigor is necessary but not sufficient to ensure high-quality research. Rig-
orous research is not necessarily “good” research. As one of our peer reviewers pointed out,
research must also be evaluated on the basis of its relevance to the profession and its poten-
tial impact on social justice. (p. 18)
We note that the distinction between impact and potential impact is an important one, since potential
impact may be assessed (perhaps inaccurately) during the course of the research, whereas actual im-
pact can only be known after the research has been disseminated.
We concede that the issue of rigor/relevance sufficiency remains muddy in our review. A major con-
tributor to this is the fuzziness of the two constructs. The situation is particular troublesome in the
(frequent) cases where relevance seems to be implicitly defined as everything that contributes to a
desired research outcome that is not explicitly addressed by rigor. For example:
Relevance is the degree to which research contributes directly to improving outcomes of in-
terest to practitioners in the field (Nunamaker et al., p. 41)

What is Research Rigor?
66
Defined in this manner, there is no need for any constructs beyond rigor and relevance to achieve the
desired research impact. Of course, such a definition also calls into question the need for “rele-
vance,” since it is essentially synonymous with “improved outcomes to practitioners in the field”.
With concerns such as those relating to research effectiveness and sufficiency of rigor and relevance
in mind, we now turn to looking at rigor as it is treated within the informing science transdiscipline.
RIGOR IN THE INFORMING SCIENCE TRANSDISCIPLINE
The previously mentioned challenges in applying the concept of rigor across a wide range of subject
areas and methodologies are particularly pronounced in informing science. By its very transdiscipli-
nary nature, the expectation is that ideas will be drawn from an extremely diverse collection of in-
forming-related research. Can the process of integrating such research ever be considered truly rigor-
ous?
DEFINING RIGOR IN INFORMING SCIENCE
The definition of rigor that has been proposed for informing science is as follows:
In order for research to be rigorous it must:
1.
Be systematic in its inquiry:
In the case of research intended to build theory, this implies that de-
termining the boundaries of what we observe is as critical as understanding the phenomenon within those
boundaries. In the case of research intended to generate detailed observations (upon which theory might
later be based), this further implies attempting to gather all information that could be relevant to the
phenomenon being observed.
2.
Employ appropriate design:
To be appropriate, a design must ensure that the methods being em-
ployed are not prone to either errors of commission (where a false relationship is detected, commonly re-
ferred to as Type I error) or errors of omission (where a significant relationship is omitted, known as
Type II error).
3.
Ask challenging questions:
If you know the answers to your questions in advance, then there is
little risk that your hypotheses will be proven incorrect. If there is no doubt of the outcome of your re-
search before you conduct it, then whatever tests are performed cannot be considered particularly stringent.
(Gill, 2016b, p. 122)
This criteria-based definition was adapted from an earlier definition originally proposed in the field of
nursing (see Allison & Rootman, 1996, p. 334). Like many of the definitions seeking to extend rigor
to non-positivist research, it is built on general criteria for assessing rigor that can be adapted to a
broad range of contexts. Its first two elements, systematic inquiry and appropriate design, seem quite
consistent with the definitions we surveyed. Words such as method, methodology, appropriate, thor-
ough, careful, exhaustive, and systematic all appear in the word frequency list.
Where the informing science definition diverges somewhat is with respect to its final criterion: asking
challenging questions. That aspect of the definition was deemed appropriate for informing science
for two reasons. First, difficulties encountered whenever interdisciplinary research is conducted
would tend to limit its application to questions for which standard disciplinary research fails to pro-
vide answers. Such questions would, almost by necessity, be challenging. Second, rigor comes with a
price, in time, effort, and constraints placed upon the researcher with respect to methodology and the
types of questions being asked. What makes a research question challenging is most likely the fact
that its possible answers are either unknown or that they stand a high likelihood of conflicting with
what is already known (e.g., the conventional wisdom). In such cases, the cost of rigor is likely to be
justified by the effort. We might even argue that if a research process ignores the cost of conducting
the research entirely, then a failure of rigor in selecting an appropriate design is indicated.

Gill & Gill
67
RESEARCH FITNESS
In our earlier discussion of “research effectiveness”, we noted that the term was intentionally vague.
Within informing science, we are particularly interested in both the durability of ideas and the degree
to which they spread throughout an informing system. For this reason, to concept of fitness—drawn
from evolutionary biology—has been proposed as a particularly appropriate construct for judging
research success (Gill, 2016a).
Briefly stated, fitness consists of two components: an entity’s potential to survive and an entity’s po-
tential to reproduce (completely or in-part). In the context of design, the entity is an artifact that may
also be broken up into a series of elements. Fitness would reflect the degree to which the artifact re-
mains in use over time and the degree to which its elements are incorporated into subsequent designs
(Gill & Hevner, 2013).
Generalizing the concept of fitness to research, the artifact is the research paper and its elements are
the distinct ideas that the paper communicates. From an academic standpoint, a reasonable way of
estimating an article’s fitness is through metrics such as citation rate. Where we are concerned with a
research artifact’s fitness with respect to practice—recognizing that the research may not be rendered
in the form of an article, but might instead be in the form of a book, a software application, or even a
presentation—we base our fitness estimates on values such as the rate at which the various ideas dif-
fuse to practice. This closely corresponds to the notion of a meme (Dawkins, 1976).
The advantage of using fitness as a measure of research effectiveness is that it can, at least in theory,
be estimated objectively. Other plausible research outcomes—such as quality and impact—will nec-
essarily be subjective. Any author or editor who has had to deal with conflicting peer reviews will be
able to attest to that. The principal drawback of using fitness to assess research outcome is that ob-
jectively untrue or verifiably falsified research can, at least in the short term, exhibit high fitness.
Some well-known examples include the study linking autism and vaccination and the twin studies
conducted by Cyril Burt. It is precisely to avoid “high fitness” research products such as these that
rigor needs to be an important contributor to fitness.
RIGOR AND RELEVANCE IN INFORMING SCIENCE
In the absence of a strong consensus, researchers will likely define rigor and relevance to be whatever
is most convenient. The best we can hope for is that they specify how they are using the terms.
Informing-based definitions
Within the informing science transdiscipline we recommend using definitions that are grounded in
existing informing science conceptual schemes. One example of such a scheme is the single client
resonance model (Gill, 2016a), shown in Figure 10, based on an earlier bias filter model (Jamieson &
Hyland, 2006). The model proposes that in order to be absorbed by a client, a message must pass
through a series of filters—not necessarily in any particular order. These filters can, in turn, either
transform the message, distort the message, or intercept it altogether. In any of these events, the in-
tended meaning of the message is not faithfully conveyed.
What we propose is that to make the terms rigor and relevance more precise when we use them in
informing science, we attempt to match them to specific filters. For relevance, the motivation filter
would be an obvious candidate. Defined in this way, relevance would be achieved if the research ad-
dresses a question or problem that is important to the client. This is consistent with some of the ex-
isting definitions (e.g., Dodge et al., 2005) but is considerably narrower than others, particularly those
that seem to define relevance as anything that is not rigor.

What is Research Rigor?
68
Figure 10: Single client resonance model (Gill, 2016a, p. 268)
Rigor presents a more challenging case. The cognitive filter deals with whether the incoming infor-
mation seems to make sense and the risk/time filter addresses the client’s reaction to uncertainty.
Both of these are elements of credibility, and therefore clearly map to rigor. They are also consistent
with the first two items in the informing science definition: appropriate design and systematic in-
quiry. Another possible filter is the information filter, which addresses whether the incoming mes-
sage contains information that is already known (or appears, at first glance, to be known). We would
argue that this can be treated as consistent with the existing definition’s third element: challenging
questions. The rationale is that in order to be considered rigorous, research should begin the process
by focusing on questions for which the answer is not already known; in other words, research con-
ducted specifically to confirm findings that are already well established is not rigorous. While this
sounds obvious, when rigor is defined solely in terms of compliance to a well-accepted method, re-
searchers may experience a strong temptation to trod well-worn paths in order to have a high likeli-
hood of finding significant results.
A simple research fitness model
If we accept the mapping of rigor and relevance to the Figure 10 model just proposed, it becomes
evident that the two constructs do not address all the proposed filters. Specifically:
• The channel filter represents the potential of the medium itself to distort the message. For ex-
ample, messages received though social media are likely to be absorbed differently from
message presented in journal articles.
• The attention filter specifies whether we are attending the channel through which a massage is
being communicated.
• The visceral filter addresses the impact of our emotional state on how we interpret the mes-
sage (or choose to ignore it).
Within informing science, we have chosen to group these three characteristics together and refer to
them as resonance. The concept is similar to stickiness (Gladwell, 2000), which is readily achieved when

Gill & Gill
69
a message is simple, unexpected, concrete, credible, emotional and tells a story (Heath & Heath,
2007). The proposed model for research fitness is presented in Figure 11.
Figure 11: Rigor, relevance and resonance model of research fitness
CONCLUSIONS
Research rigor means different things to different people. The review that we have presented here
will not change this. Our hope, however, is that it will encourage researchers not to use the term off-
handedly. Too often we have observed the term being used casually in the literature, seemingly as-
suming that the reader perceives it to mean the same as the author.
For a transdiscipline such as informing science, the challenge presented by rigor is particularly acute.
Even within disciplines there are disputes about what is rigorous. Across disciplines the gap is huge.
For this reason, we encourage the use of flexible, criteria-based definitions of rigor that can be inter-
preted across a wide range of research contexts. Definitions based on the degree to which accepted
procedures are followed tend to be quite narrow with respect to where they can be applied. Moreo-
ver, they can easily focus the researcher’s entire attention on following the procedure instead of ask-
ing whether or not the procedure, along with the question it is intended to address, is sensible.
For informing science, our research suggests that the existing proposed definition of rigor is plausi-
ble. It is a criteria-based definition that views research rigor in terms of three characteristics:
1. It is systematic in its inquiry.
2. It employs an appropriate design for the questions being asked.
3. It asks challenging questions.
Defined in this way, rigor is clearly only part of what makes for effective research. We caution the
reader against grouping everything else needed to make research impactful into “relevance.” While
we observed that such an approach was common in the literature, we believe that it forces many un-
related aspects of research together. Instead, we advocate using a conceptual scheme, such as the
single client resonance model (Gill, 2016a, p. 10) to develop theory-based distinctions. For the pur-
poses of informing science, we propose:
• Rigor: Is the research credible and is it challenging?
• Relevance: Are we studying a question or challenge that our clients are motivated to address?
• Resonance: Are we communicating the research in a manner that grabs the client’s attention
and engages the client emotionally using a channel appropriate to the content?

What is Research Rigor?
70
We further propose that the most effective research is that which endures over time and, most im-
portantly, diffuses to its intended clientS. For this reason, we suggest that achieving research fitness
should be our principle objective in conducting and presenting our research.
REFERENCES
Aguinis, H., Werner, S., Lanza Abbott, J., Angert, C., Park, J. H., & Kohlhausen, D. (2010). Customer-centric
science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organiza-
tional Research Methods, 13(3), 515-539. https://doi.org/10.1177/1094428109333339
Allison, K.R. & Rootman, I. (1996) Scientific rigor and community participation in health promotion research:
are they compatible? Health Promotion International, 11(4), 333-340.
https://doi.org/10.1093/heapro/11.4.333
Applegate, L. M., & King, J. L. (1999). Rigor and relevance: Careers on the line. MIS Quarterly, 23(1), 17-18.
https://doi.org/10.2307/249404
Armour, M., Rivaux, S. L., & Bell, H. (2009). Using context to build rigor: Application to two hermeneutic
phenomenological studies. Qualitative Social Work, 8(1), 101-122.
https://doi.org/10.1177/1473325008100424
Bandara, W., Furtmueller, E., Gorbacheva, E., Miskon, S., & Beekhuyzen, J. (2015). Achieving rigor in litera-
ture reviews: Insights from qualitative data analysis and tool-support. Communications of the Association for In-
formation Systems, 37, 154-204. https://doi.org/10.17705/1cais.03708
Barbour, R. S. (2001). Checklists for improving rigour in qualitative research: A case of the tail wagging the
dog? BMJ, 322(7294), 1115-1117. https://doi.org/10.1136/bmj.322.7294.1115
Barusch, A., Gringeri, C., & George, M. (2011). Rigor in qualitative social work research: A review of strategies
used in published articles. Social Work Research, 35(1), 11-19. https://doi.org/10.1093/swr/35.1.11
Biggs, M. A., & Büchler, D. (2007). Rigor and practice-based research. Design Issues, 23(3), 62-69.
https://doi.org/10.1162/desi.2007.23.3.62
Collier, D., LaPorte, J., & Seawright, J. (2012). Putting typologies to work: Concept formation, measurement,
and analytic rigor. Political Research Quarterly, 65(1), 217-232. https://doi.org/10.1177/1065912912437162
Combs, J. G. (2010). Big samples and small effects: Let’s not trade relevance and rigor for power. Academy of
Management Journal, 53(1), 9-13. https://doi.org/10.5465/amj.2010.48036305
Daft, R. L., & Lewin, A. Y. (2008). Perspective—Rigor and relevance in organization studies: Idea migration
and academic journal evolution. Organization Science, 19(1), 177-183.
https://doi.org/10.1287/orsc.1070.0346
Darke, P., Shanks, G., & Broadbent, M. (1998). Successfully completing case study research: Combining rigour,
relevance and pragmatism. Information Systems Journal, 8(4), 273-289. https://doi.org/10.1046/j.1365-
2575.1998.00040.x
Davenport, T. H., Markus, M. L., & Lynne, M. (1999). Rigor vs. relevance revisited. MIS Quarterly, 23(1), 19-23.
https://doi.org/10.2307/249405
Dawkins, R. (1976). The selfish gene. Oxford University Press.
Dodge, J., Ospina, S. M., & Foldy, E. G. (2005). Integrating rigor and relevance in public administration schol-
arship: The contribution of narrative inquiry. Public Administration Review, 65(3), 286-300.
https://doi.org/10.1111/j.1540-6210.2005.00454.x
Dubé, L., & Paré, G. (2003). Rigor in information systems positivist case research: Current practices, trends,
and recommendations. MIS Quarterly, 27(4), 597-636. https://doi.org/10.2307/30036550
Eisenhardt, K. M. (1991). Better stories and better constructs: The case for rigor and comparative logic. Acade-
my of Management Review, 16(3), 620-627. https://doi.org/10.5465/amr.1991.4279496

Gill & Gill
71
Eisenhardt, K. M., Graebner, M. E., & Sonenshein, S. (2016). Grand challenges and inductive methods: Rigor
without rigor mortis. Academy of Management Journal, 59(4), 1113–1123.
https://doi.org/10.5465/amj.2016.4004
Erickson, F., & Gutierrez, K. (2002). Comment: Culture, rigor, and science in educational research. Educational
Researcher, 31(8), 21-24. https://doi.org/10.3102/0013189x031008021
Fereday, J., & Muir-Cochrane, E. (2006). Demonstrating rigor using thematic analysis: A hybrid approach of
inductive and deductive coding and theme development. International Journal of Qualitative Methods, 5(1), 80-
92. https://doi.org/10.1177/160940690600500107
Frank, H., & Landström, H. (2016). What makes entrepreneurship research interesting? Reflections on strate-
gies to overcome the rigour-relevance gap. Entrepreneurship & Regional Development, 28(1-2), 51-75.
https://doi.org/10.1080/08985626.2015.1100687
Frauenberger, C., Good, J., Fitzpatrick, G., & Iversen, O. S. (2015). In pursuit of rigour and accountability in
participatory design. International Journal of Human-Computer Studies, 74, 93-106.
https://doi.org/10.1016/j.ijhcs.2014.09.004
Gasson, S. (2004). Rigor in grounded theory research: An interpretive perspective on generating theory from
qualitative field studies. In M. Whitmann & A. Woszczynski (Eds.), The handbook of information systems re-
search (pp. 79-102). IGI Global. https://doi.org/10.4018/978-1-59140-144-5.ch006
Gibbert, M., & Ruigrok, W. (2010). The ‘‘what’’ and ‘‘how’’ of case study rigor: Three strategies based on pub-
lished work. Organizational Research Methods, 13(4), 710-737. https://doi.org/10.1177/1094428109351319
Gill, T. G. (2016a) Informing science, volume 1: Concepts and systems. Informing Science Press.
Gill, T. G. (2016b) Informing science, volume 2: Design and research issues. Informing Science Press.
Gill, T. G. & Hevner, A. R. (2013). A fitness-utility model for design science research. ACM Transactions on
Management Information Systems, 4(2), 5:1-5:24. https://doi.org/10.1145/2499962.2499963
Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2013). Seeking qualitative rigor in inductive research: Notes on
the Gioia methodology. Organizational Research Methods, 16(1), 15-31.
https://doi.org/10.1177/1094428112452151
Gladwell, M. (2000). The tipping point. Back Bay Books.
Grover, V., Lee, C. C., & Durand, D. (1993). Analyzing methodological rigor of MIS survey research from
1980–1989. Information & Management, 24(6), 305-317. https://doi.org/10.1016/0378-7206(93)90028-r
Gulati, R. (2007). Tent poles, tribalism, and boundary spanning: The rigor-relevance debate in management
research. Academy of Management Journal, 50(4), 775-782. https://doi.org/10.5465/amj.2007.26279170
Gutiérrez, K. D., & Penuel, W. R. (2014). Relevance to practice as a criterion for rigor. Educational Researcher,
43(1), 19-23. https://doi.org/10.3102/0013189x13520289
Hasson, F., & Keeney, S. (2011). Enhancing rigour in the Delphi technique research. Technological Forecasting and
Social Change, 78(9), 1695-1704. https://doi.org/10.1016/j.techfore.2011.04.005
Heath, C., & Heath, D. (2007). Made to stick. Random House
Hodgkinson, G. P., & Rousseau, D. M. (2009). Bridging the rigour–relevance gap in management research: It’s
already happening! Journal of Management Studies, 46(3), 534-546. https://doi.org/10.1111/j.1467-
6486.2009.00832.x
Houston, M. B. (2019). Four facets of rigor. Journal of the Academy of Marketing Science, 47, 570–573
https://doi.org/10.1007/s11747-019-00665-7
Ivarsson, M., & Gorschek, T. (2011). A method for evaluating rigor and industrial relevance of technology
evaluations. Empirical Software Engineering, 16(3), 365-395. https://doi.org/10.1007/s10664-010-9146-4
Jamieson, K., & Hyland, P. (2006). Good intuition or fear and uncertainty: The effects of bias on information
systems selection decisions. Informing Science: The International Journal of an Emerging Transdiscipline, 9, 49-69.
https://doi.org/10.28945/471

What is Research Rigor?
72
Kieser, A., & Leiner, L. (2009). Why the rigour–relevance gap in management research is unbridgeable. Journal
of Management Studies, 46(3), 516-533. https://doi.org/10.1111/j.1467-6486.2009.00831.x
Kieser, A., & Nicolai, A. T. (2005). Success factor research: Overcoming the trade-off between rigor and rele-
vance? Journal of Management Inquiry, 14(3), 275-279. https://doi.org/10.1177/1056492605279098
Kincheloe, J. L. (2011). Describing the bricolage: Conceptualizing a new rigor in qualitative research. In k.
hayes, S. R. Steinberg & K. Tobin (Eds.), Key works in critical pedagogy (pp. 177-189). Brill Sense.
https://doi.org/10.1007/978-94-6091-397-6_15
Lee, A. S., & Hubona, G. S. (2009). A scientific basis for rigor in information systems research. MIS Quarterly,
33(2), 237-262. https://doi.org/10.2307/20650291
Lu, C. J., & Shulman, S. W. (2008). Rigor and flexibility in computer-based qualitative research: Introducing the
Coding Analysis Toolkit. International Journal of Multiple Research Approaches, 2(1), 105-117.
https://doi.org/10.5172/mra.455.2.1.105
McAlister, L. (2016). Rigor versus method imperialism. Journal of the Academy of Marketing Science, 44(5), 565-567.
Melrose, M. J. (2001). Maximizing the rigor of action research: Why would you want to? How could you? Field
Methods, 13(2), 160-180. https://doi.org/10.1177/1525822x0101300203
Nowell, B., & Albrecht, K. (2018). A reviewer’s guide to qualitative rigor. Journal of Public Administration Research
and Theory, 29(2), 348-363. https://doi.org/10.1093/jopart/muy052
Nunamaker, J. F., Jr., Briggs, R. O., Derrick, D. C., & Schwabe, G. (2015). The last research mile: Achieving
both rigor and relevance in information systems research. Journal of Management Information Systems, 32(3),
10-47. https://doi.org/10.1080/07421222.2015.1094961
O’Dwyer, B., & Unerman, J. (2016). Fostering rigour in accounting for social sustainability. Accounting, Organiza-
tions and Society, 49, 32-40. https://doi.org/10.1016/j.aos.2015.11.003
Ogawa, R. T., & Malen, B. (1991). Towards rigor in reviews of multivocal literatures: Applying the exploratory
case study method. Review of Educational Research, 61(3), 265-286.
https://doi.org/10.3102/00346543061003265
Onwuegbuzie, A. J., Dickinson, W. B., Leech, N. L., & Zoran, A. G. (2009). Toward more rigor in focus group
research: A new framework for collecting and analyzing focus group data. International Journal of Qualitative
Methods, 8(3), 1-21. https://doi.org/10.1177/160940690900800301
Open Science Collaboration. (2015, August 28). Estimating the reproducibility of psychological science, Science,
349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Poland, B. D. (1995). Transcription quality as an aspect of rigor in qualitative research. Qualitative Inquiry, 1(3),
290-310. https://doi.org/10.1177/107780049500100302
Rajagopalan, N. (2019). Rigor, relevance, and resilience in management research. Journal of Management Inquiry,
29(2), 150–153. https://doi.org/10.1177/1056492619861690
Rand, W., & Rust, R. T. (2011). Agent-based modeling in marketing: Guidelines for rigor. International Journal of
Research in Marketing, 28(3), 181-193. https://doi.org/10.1016/j.ijresmar.2011.04.002
Robey, D., & Markus, M. L. (1998). Beyond rigor and relevance: Producing consumable research about infor-
mation systems. Information Resources Management Journal (IRMJ), 11(1), 7-16.
https://doi.org/10.4018/irmj.1998010101
Rosas, S. R., & Kane, M. (2012). Quality and rigor of the concept mapping methodology: A pooled study analy-
sis. Evaluation and Program Planning, 35(2), 236-245. https://doi.org/10.1016/j.evalprogplan.2011.10.003
Ross, S. M., Morrison, G. R., & Lowther, D. L. (2010). Educational technology research past and present: Bal-
ancing rigor and relevance to impact school learning. Contemporary Educational Technology, 1(1), 17-35.
https://doi.org/10.30935/cedtech/5959
Sculley, D., Snoek, J., Wiltschko, A., & Rahimi, A. (2018, April 30-May 3). Winner’s curse? On pace, progress,
and empirical rigor [Workshop paper]. 6th International Conference on Learning Representations (ICLR 2018),
Vancouver Convention Center, Vancouver, BC, Canada.

Gill & Gill
73
Seuring, S. A. (2008). Assessing the rigor of case study research in supply chain management. Supply Chain Man-
agement: An International Journal, 13(2), 128-137. https://doi.org/10.1108/13598540810860967
Shrivastava, P. (1987). Rigor and practical usefulness of research in strategic management. Strategic Management
Journal, 8(1), 77-92. https://doi.org/10.1002/smj.4250080107
Snow, C. E. (2015). 2014 Wallace Foundation distinguished lecture: Rigor and realism: Doing educational sci-
ence in the real world. Educational Researcher, 44(9), 460-466. https://doi.org/10.3102/0013189x15619166
Sovacool, B. K., Axsen, J., & Sorrell, S. (2018). Promoting novelty, rigor, and style in energy social science: to-
wards codes of practice for appropriate methods and research design. Energy Research & Social Science, 45,
12-42. https://doi.org/10.1016/j.erss.2018.07.007
Stewart, H., Gapp, R., & Harwood, I. (2017). Exploring the alchemy of qualitative management research: Seek-
ing trustworthiness, credibility and rigor through crystallization. The Qualitative Report, 22(1), 1-19.
Tushman, M. L., O’Reilly, C., Fenollosa, A., Kleinbaum, A. M., & McGrath, D. (2007). Relevance and rigor:
Executive education as a lever in shaping practice and research. Academy of Management Learning & Educa-
tion, 6(3), 345-362. https://doi.org/10.5465/amle.2007.26361625
Van Weele, A. J., & Van Raaij, E. M. (2014). The future of purchasing and supply management research: About
relevance and rigor. Journal of Supply Chain Management, 50(1), 56-72. https://doi.org/10.1111/jscm.12042
Varadarajan, P. R. (2003). Musings on relevance and rigor of scholarly research in marketing. Journal of the Acad-
emy of Marketing Science, 31(4), 368-376. https://doi.org/10.1177/0092070303258240
Vermeulen, F. (2005). On rigor and relevance: Fostering dialectic progress in management research. Academy of
Management Journal, 48(6), 978-982. https://doi.org/10.5465/amj.2005.19573102
Walt, S. M. (1999). Rigor or rigor mortis? Rational choice and security studies. International Security, 23(4), 5-48.
https://doi.org/10.1162/isec.23.4.5
Webster, F. E., Jr. (1978). Management science in industrial marketing: A review of models and measurement
techniques–new rigor, new sophistication. Journal of Marketing, 42(1), 21-27.
https://doi.org/10.1177/002224297804200106
White, S. (2002). Rigor and relevance in Asian management research: Where are we and where can we go? Asia
Pacific Journal of Management, 19(2-3), 287-352.

What is Research Rigor?
74
APPENDIX: DEFINITIONS OF RIGOR
Source
Definitional Quotes
Armour et al. (2009)
Rigor is the degree to which researchers hold themselves to standards of in-
quiry that address challenges to the credibility of a study’s findings. Rigor,
therefore, can refer either or both to methodological thoroughness and pre-
cision or criteria used to judge the trustworthiness of the results. Methodo-
logical stringency and accuracy of the results are related because solidity in
methods provides greater assurance that the findings are valid. (p. 102)
Biggs & Büchler
(2007)
Rigor refers to the process of undertaking activities such as the literature
search. It connotes a systematic and thorough search. As a result, the re-
searcher can be confident that from a “null return” (i.e., when the research-
er’s knowledge and understanding has been identified as absent from the
published body of knowledge and understanding in the field), it can be con-
cluded that the researcher’s knowledge and understanding is new knowledge
and understanding. (p. 66)
In conclusion, rigor in research is the strength of the chain of reasoning, and
that has to be judged in the context of the question and the answer, for ex-
ample, in the context of design as opposed to the context of physics or phi-
losophy (p. 69)
Dodge et al. (2005)
As a starting point, rigor traditionally refers to the accurate and systematic
application of theory and method (p. 288)
Interpretive rigor goes beyond the sound application of method to focus on
moral and aesthetic dimensions of scholarly quality. It pushes researchers to
engage strategies that directly contribute to worthwhile human purposes, not
only from their own perspective, but from the perspectives of other social
actors involved in the research (p. 297)
Fereday & Muir-
Cochrane (2006)
Rigor is described as demonstrating integrity and competence within a study.
... Schutz’s first postulate of logical consistency is similar to the description
by Horsfall, Byrne-Armstrong, and Higgs of rigor in qualitative research,
which involves in-depth planning, careful attention to the phenomenon un-
der study, and productive, useful results. Descriptions of theoretical rigor
involve sound reasoning and argument and a choice of methods appropriate
to the research problem. ... The step-by-step process of analysis that is out-
lined in this article is a method of demonstrating transparency of how the
researcher formulated the overarching themes from the initial participant
data. ... Interpretive rigor requires the researcher to demonstrate clearly how
interpretations of the data have been achieved and to illustrate findings with
quotations from, or access to, the raw data (p.82)
Frauenberger et al.
(2015)
The notion of “rigour” is commonly associated with a strict positivistic view
on science, emphasising universal truths validated by deductive reasoning or
measured evidence (p. 94)
We argue that rigorous PD [participatory design] work exhibits qualities that
are coherent, e.g., it is based on an epistemology that accommodates the val-
ues that drive the effort, involves stakeholders in ways that reflect these
foundations and accordingly defines and delivers its meaningful outcomes.
(p. 103)
Gulati (2007)
Sometimes scholars define rigor as use of a narrow disciplinary paradigm
involving a set of theories, methodologies, and data analyses that they them-
selves would use (p. 777)

Gill & Gill
75
Source
Definitional Quotes
Hasson & Keeney
(2011)
The “holy grail” of research is establishing methodological rigour. This refers
to a researcher’s responsibility to ensure that procedures have been adhered
to and confounding factors eliminated [where possible] to produce dependa-
ble results. (p. 1695)
Houston (2019)
Rigor is not just a buzzword—it is the basis for having confidence in re-
search findings. In the words of Miriam-Webster’s Dictionary, it is the quali-
ty of being extremely thorough, exhaustive, or accurate; the English Lan-
guage Learner’s Dictionary defines it as the quality or state of being very ex-
act, careful, or strict. Applied to research, these definitions imply that rigor is
more than the application of sophisticated and complicated quantitative
methodologies. (p. 570)
[Four facets of rigor]
Facet 1: Rigor in designing research questions...
Facet 2: Conceptual rigor...
Facet 3: Methodological and analytical rigor...
Facet 4: Rigor in crafting a scholarly manuscript (pp. 571-572)
Ivarsson & Gorschek
(2011)
Rigor refers to both how an evaluation is performed and how it is reported.
If the study is not adequately described, the rigor of the evaluation cannot be
evaluated by reviewers and other researchers. (p. 367)
Rigor in research often refers to the precision or exactness of the research
method used; e.g. a controlled experiment often enables greater control over
variables than a case study. ...This is one way to view rigor, the precision of
the research approach utilized. Rigor can also mean the correct use of any
method for its intended purpose ... implying that there is a context or appli-
cation in which certain methods are appropriate or applicable. (p. 369)
McAlister (2016)
Rigorous (defined as extremely thorough, exhaustive, or accurate) does not
imply newness or complicatedness. Rigorous explicitly includes accuracy, and
it implies that one has chosen the right method for the problem. (p. 565)
Melrose (2001)
One term that was used at this conference by some presenters as synony-
mous with rigor was validity ... others used the term in a broader sense, refer-
ring to the whole process or to different parts of the process of research
(e.g., the choice of method to suit the research question, the constitution and
facilitation of collaborative research groups, and the dissemination of the
data in appropriate ways to suit audiences) (p. 163)
Another meaning for rigorous is “scrupulous” ... Here the idea of rigor is
linked to telling the truth (as far as can be established by evidence) and be-
having ethically. …One common definition of ethical behavior is behavior
that does no one harm. (pp. 174-175)
In addition to the meanings explored so far in this article, rigor can also
mean “not bending or inflexible,” “harsh or oppressive,” and “very strict” (p.
177)
Nowell & Albrecht
(2018)
Rigor, then, can be conceptualized as the appropriate execution of that
method. Put simply, if quality is the what, rigor for our purposes becomes
the how. (p. 352)
rigorous analysis is based on 1) whether the interpretation is credible in light
of the data, 2) whether it was the result of a robust and systematic analytical
process designed to move beyond superficial findings and minimize and/or
account for investigator bias, and 3) whether it is reported with sufficient
attention to context so as to facilitate the potential relevance of insights to
similar contexts (p. 357)

What is Research Rigor?
76
Source
Definitional Quotes
Nunamaker et al.
(2015)
Rigor is the degree to which research practices follow the standards of logic
dictated by the epistemology under which it claims to have contributed new
knowledge (p. 41)
Ogawa & Malen
(1991)
In simplified terms, rigor involves adherence to principles and procedures,
methods, and techniques that minimize bias and error in the collection, anal-
ysis, interpretation, and reporting of data. (p. 267)
Sovacool et al. (2018)
research also needs to improve in terms of rigor (depth), interdisciplinary
reach (breadth), policy-relevance, and the communication of results (p. 13)
criticisms have also been levied at the lack of rigor in academic research. By
this, we mean a mix of carefulness and thoroughness. The simple Oxford
definition of rigor is “the quality of being extremely thorough and careful.”
This definition does not favor a particular research design, objective, disci-
pline or method. Rather, this definition represents the practice of taking great
care in establishing and articulating research objectives, selecting and imple-
menting appropriate research methods and interpreting research results -
while at the same time acknowledging omissions and limitations (p. 13)
In this way, our definition of rigor is about being “careful and thorough” in
one’s research, but not necessarily using the most advanced, sophisticated or
complicated method. All methods have their strengths and limitations, so an
effective definition of rigor is more of a “good balance across multiple crite-
ria.” (p. 32)
BIOGRAPHIES
Grandon Gill is a professor in the Information Systems and Decision
Sciences Department of the University of South Florida. He is also the
Academic Director of the Doctor of Business Administration Program at
the Muma College of Business. He is Editor-in-Chief of the Muma Busi-
ness Review and the past Editor-in-Chief of Informing Science: The In-
ternational Journal of an Emerging Transdiscipline and the Journal of IT
Education: Discussion Cases, also serving as a Governor and Fellow of
the Informing Science Institute, where he was elected President in 2019.
He was also the inaugural recipient of the Zbigniew Gackowski Award
for informing science research
Thomas R. Gill is a doctoral student in the Information Systems and
Decision Sciences Department at the University of South Florida. He has
a Bachelor of Science in Computer Science from the College of William
and Mary and a Master of Science in Business Analytics and Information
Systems from the University of South Florida. He co-authored an article
published in Cancer Informatics.