ArticlePDF Available

Abstract

We introduce a method in which instant data visualization facilitates real-time data integration and involves participants in data interpretation. The results of quantitative research (e.g., electronic card sorting) are represented visually (e.g., in a dendrogram) and fed back to research participants in follow-up focus group conversations. The visualized quantitative results are reviewed and discussed by participants. The visual display of the quantitative results is annotated with qualitative feedback generated by participants that explains, enriches, or challenges the quantitative results. We apply our method in a card sorting study of Fédération Internationale de Football Association’s (FIFA) stakeholders. An approach that facilitates real-time data integration that is participant driven and visually supported is the unique contribution of this article to mixed methods research.
REAL-TIME INTEGRATION 1
Elitsa Alexander , Martin J. Eppler1 and Alice Comi (2020)
Data Integration: A Real- Time, Participant-Driven, and Visually Supported Method
Journal of Mixed Methods Research Methods Research (online first).
Available online at: https://journals.sagepub.com/doi/pdf/10.1177/1558689820902294
HOW TO CITE THIS ARTICLE:
Alexander, E., Eppler, M. J., & Comi, A. (2020). Data Integration: A Real- Time,
Participant-Driven, and Visually Supported Method. Journal of Mixed Methods Research, online
first, 1–27. http://doi.org/10.1177/1558689820902294
REAL-TIME INTEGRATION 2
Data Integration: a Real-time, Participant-driven, and Visually Supported Method
REAL-TIME INTEGRATION 3
Abstract
We introduce a method in which instant data visualization facilitates real-time data integration
and involves participants in data interpretation. The results of quantitative research (e.g.,
electronic card sorting) are represented visually (e.g., in a dendrogram) and fed back to research
participants in follow-up focus group conversations. The visualized quantitative results are
reviewed and discussed by participants. The visual display of the quantitative results is annotated
with qualitative feedback generated by participants that explains, enriches, or challenges the
quantitative results. We apply our method in a card sorting study of Fédération Internationale de
Football Association’s (FIFA) stakeholders. An approach that facilitates real-time data integration
that is participant-driven and visually supported is the unique contribution of this article to mixed
methods research.
Keywords: real-time integration; collaborative interpretation; visual representation
REAL-TIME INTEGRATION 4
Integration: a Real-time, Participant-driven, and Visually Supported Method
Background and Methodological Purpose
In mixed methods research, researchers combine elements of qualitative and quantitative
approaches (e.g. qualitative and quantitative viewpoints, data collection, analysis, and inference
techniques) to gain breadth and depth in understanding and corroboration (Johnson,
Onwuegbuzie, & Turner, 2007, p. 123). According to a recent definition, mixed methods include
“any research that involves multiple sources and types of data and/or multiple approaches to
analysis of those data, in which integration of data and analyses occurs prior to drawing final
conclusions about the topic of the investigation” (Bazeley, 2018, p. 7). Notably, “meaningful
integration of qualitative and quantitative data remains elusive and needs further development’’
(Guetterman, Fetters, & Creswell, 2015, p. 554). Researchers employing mixed methods often
experience difficulty in integrating the analysis and interpretation of the quantitative and the
qualitative data and writing a narrative that link[s] the analyses and interpretations (Bryman,
2007, p. 10).
Hence, there is a need to find ways to facilitate meaningful integration of quantitative and
qualitative data. Our attention turned specifically to “development of descriptive and narrative
accounts from quantitative (statistical) data” (Bazeley, 2018, p. 223). Such accounts result
whenever researchers describe, interpret, or discuss their findings in prose. As integration of this
kind is not often used as a deliberate analysis strategy in mixed methods projects, it may offer
“untapped potential” (Bazeley, 2018, p. 223). In this article we unleash this potential by offering
REAL-TIME INTEGRATION 5
a visual approach to elicitation of qualitative participant interpretations of visualized quantitative
data in a real-time data integration. The methodological purpose of our article is to provide a
method which facilitates real-time integration that is participant-driven and visually supported.
Our work is structured as follows. We begin with a review of the literature on the use of
visual representations in mixed methods research and on participant involvement in data
interpretation. We then provide a summary of our method, including a description of its
constitutive phases and their outcomes, as well as a description of an illustrative study in which
the method was employed. Finally, we evaluate the method and discuss it with reference to its
contribution and limitations. The following section aims to clarify what types of visual
representations exist and to contextualize their role in mixed methods research.
Literature Review
Use of Visual Representations in Mixed Methods Research
Building on recent efforts to systematize the use of visual representations in mixed
methods research (Archibald, 2018; Balomenou & Garrod, 2015; D’Angelo, Ryan, & Tubaro,
2016; Guetterman, Fetters, and Creswell, 2015; Onwuegbuzie & Dickinson, 2008; Shannon-
Baker & Edwards, 2018), it is possible to identify three main (often overlapping) purposes of use.
Visual representations are used in mixed methods research for data elicitation, data integration
and interpretation, and data communication.
First, mixed methods researchers use visual representations to elicit that which is difficult
to verbalize or observe. Visual representations (e.g., diagrams, drawings) are produced by
researchers or by the research participants. Visual representations that are produced by
researchers are used as stimuli in the study to elicit responses from research participants. For
example, Tubaro et al. (2016, p. 7) asked participants to fill in the blanks on a diagrammatic
REAL-TIME INTEGRATION 6
representation of concentric circles and Alexander et al. (2015, p. 38) asked participants to fill in
the blanks on a metaphoric representation of a funnel. Visual representations that are produced
by the research participants express feelings or illustrate situations introspectively and
reflectively. For example, the participants in the studies of Brechet et al. (2009), O’Connell
(2013), and Shannon-Baker (2015) produced self-portraits, drawings and photos to express
feelings and illustrate situations.
Second, mixed methods researchers use visual representations to integrate and interpret
quantitative and qualitative data in order to derive new insights beyond the information gained
from the separate quantitative and qualitative results (Fetters, Curry, & Creswell, 2013, p. 2143)
and hence to enhance their understanding of the phenomena under analysis (Onwuegbuzie &
Dickinson, 2008). For example, mixed methods researchers use joint displays (Guetterman et al.,
2015) to integrate quantitative and qualitative data by bringing them together with the purpose of
direct comparison.
Third, mixed methods researchers use visual representations to communicate research
results to the readers. For example, crossover graphical displays (Onwuegbuzie & Dickinson,
2008) summarize integrated quantitative and qualitative results in (interactive) line charts,
georeferencing plots, bubble plots, scatterplots, pictograms, maps, and (decision) trees. The title
“crossover” (Onwuegbuzie & Combs, 2010) comes from using techniques from one tradition
(e.g., quantitative) to analyze data associated with the other tradition (qualitative).
Visually supported mixed methods studies can be conceptualised as involving different
levels of visual representations. In his seminal work, Tufte identified five levels of visual
representations (Tufte, 2001, p. 178): text (level 1); tables (level 2), which show exact numerical
values; text-tables (level 3), which summarize and arrange numeric data by type (i.e.,
REAL-TIME INTEGRATION 7
demographics, source, time, group membership, scale, level) to facilitate comparison;
supertables (level 4), which provide organized, sequential detail, and reference-like quality and
may contain pictures; and graphics (level 5), which combine words, numbers, and pictures.
The visual representation used in Nicca et al. (2012, p. 229) can be categorized as a text-
table – corresponding to Tufte’s third level of representations (Table 1). For the purposes of data
integration/interpretation and data communication in a multi-phase mixed methods study, this
table summarizes and arranges numeric data by “symptoms” according to a questionnaire,
“appraisal of symptoms” according to contrasting groups of participants in narratives elicited
after the questionnaire, and the results of testing hypotheses regarding differences in the
participants’ narratives.
-----------------------------------
Insert Table 1 about here
-----------------------------------
The elaborate table used at the data interpretation stage for Alexander et al. (2015) can be
categorized as a supertable – corresponding to Tufte’s fourth level of representations (Table 2).
This supertable contains numeric, textual, and visual data (i.e., the experimental conditions are
visually represented). The supertable allows comparisons between the shared and withdrawn
thoughts regarding project experiences. The horizontal rules divide the data into group-related
paragraphs; the rows are ordered so as to tell an ordered story about the experimental groups and
the individual members of the experimental groups (based on integrated thoughts). Further
examples of elaborate tables may be found in Peroff et al. (2019, p. 8) and Flowers et al. (2015, p.
851).
-----------------------------------
Insert Table 2 about here
-----------------------------------
REAL-TIME INTEGRATION 8
The diagram used in Bustamante (2019, p. 171, Table 3) can be categorized as
corresponding to Tufte’s fifth level of representations. This diagram was used to represent the
integration of quantitative and qualitative data: black represents quantitative data, white
qualitative data, and gray represents the mixing of black and white – in this case, the outcome, or
“fit”, resulting from integrating the quantitative and qualitative data. The diagram used in Tubaro
et al. (2016, p. 7) also corresponds to Tufte’s fifth level of representations, but has been used for
the purpose of data elicitation. The personal social network of each research participant (Table 3)
contains concentric circles that represent relational proximity to ego, while the quadrants
represent the context of relationships in the social network. Research participants were asked to
fill in the blanks between the circles. Similarly, Alexander et al. (2015, p. 38) used a
diagrammatic representation (a funnel metaphor) for the purpose of data elicitation. The research
participants were asked to fill in the blanks on the funnel in order to share their project-related
experiences. The types of level-5 visual representations used in mixed methods include arts-
based graphics. For example, Brechet et al. (2009), O’Connell (2013), and Shannon-Baker
(2015) collected self-portraits, drawings and photos produced by participants and analysed them
in mixed methods studies.
-----------------------------------
Insert Table 3 about here
-----------------------------------
Visual representations have the potential to add to the inference-drawing capacity of
researchers, but not necessarily to the integration of data. Bazeley (2018) points out that visual
displays generated by “software are helpful to varying degrees in revealing and displaying
patterns in data – sometimes more for the researcher [emphasis added] during the analysis
process than for the reader of a report” (p. 297). But what about for participants, who are
increasingly being drawn into the process of data analysis and interpretation?
REAL-TIME INTEGRATION 9
Involving Participants in Data Interpretation
The idea of involving research participants in data interpretation stems from early
concerns about the provisional nature of knowledge and the limits of objectivity (Popper, 1959).
Participant involvement has been motivated by early concerns regarding interpretation biases
(MacCoun, 1998) as well as by more recent concerns that inferences drawn “should make sense
to those who contributed the data” (Bazeley, 2018, p. 55). So-called reflexive (or participatory)
research methods describe a partnership between researchers and research participants in order to
use the knowledge and abilities of each (Van de Ven, 2007). Participatory methods have been
used in attempts to involve participants in data interpretation (as e.g. “co-researchers”). Such
methods are frequently used in fields like postmodern ethnography (e.g. Presnell, 1994),
anthropology (e.g. Feighery, 2006), applied communication research (e.g. Deetz, 2008), action
research (e.g. Heron & Reason, 2006; Reason & Bradbury, 2008), and less frequently used in
management and organization studies (Bartunek, 2007; Van Aken, 2004; Van de Ven, 2007). In
mixed methods, participatory research is “one expression of a pragmatist position” (Garner, 2015,
p. 179). For example, community-based participatory research (Israel et al., 2013) combined
with mixed methods research (Dejonckheere et al., 2018) engage research participants in the
design and implementation of research that may benefit society (Molina-Azorin & Fetters, 2019).
Ivankova (2014) first discussed intersecting mixed methods with action and participatory
research approaches.
It has been suggested that researchers and participants collectively negotiate the meaning
of results to help complement otherwise “incomplete” research (Alvesson et al., 2008, p. 483).
Another proposal for involving participants is to open up research texts to “multiple readings” by
participants and audiences (Alvesson et al., 2008; King & Learmonth, 2015; Lukka & Modell,
REAL-TIME INTEGRATION 10
2010). These multiple readings may take place (a) during “data collection and initial analysis”
(through events such as meetings and workshops), (b) during “interpretation” (through joint
interpretive forums), (c) during “dissemination” (through co-authorship of research reports), or
(d) during “implementation” (through guidance and advice on implementation) (Knight &
Pettigrew, 2007, pp. 7–9). Joint interpretive forums have been suggested by Mohrman et al.
(2001) and Rynes et al. (2001) as events (like workshops or conversations) where researchers
and practitioners work together to interpret results. However, both Mohrman et al. (2001) and
Rynes et al. (2001) are conceptual epistemological papers, which suggest (but do not apply)
“joint interpretive forums” as a concept.
Kieser and Leiner (2009, p. 527) noted that the overwhelming majority of articles
resulting from collaborative research [with participants] are of an epistemological kind. Kieser
and Leiner also indicated that they did not know of any publications that contain jointly
produced research output describing research results rather than processes and the difficulties of
collaboration between researchers and research participants. Hence there is a recognized need to
find ways to facilitate joint researcher-participant interpretation of research results.
According to Howe and Eisenhart (1990), Bazeley (2018, p. 56), and Kuckartz (2018),
“all scientific analysis involves acts of interpretation by researchers” [italic added]. The role of
the researcher remains paramount in deciding issues relating to ... the meaning of codes, the
interpretation of the data tables and displays produced using the computer (Bazeley, 2010, p.
418–419). Interpretation conducted by researchers is influenced by their research purposes, their
subjective awareness and sensitivity, the context in which the data were obtained, the underlying
conceptual framework, and the choice of methodology. Interpretation of data still remains largely
the prerogative of scholars, even though participants commit substantial time and resources to
REAL-TIME INTEGRATION 11
the studies and numerous approaches have set out to involve them in the process. With this work,
therefore, we aim to contribute by offering a visual approach to involve participants – by using
visual representations to elicit qualitative participant interpretations of visualized quantitative
data in a real-time data integration.
In this section, we have (1) reviewed the visual representations used in mixed methods
research, and (2) shown that there is a recognized need to find ways to facilitate joint researcher-
participant interpretation of research results. On the basis of these methodological needs, we now
introduce an innovative method enabling research participants to review quantitative results
through visual displays shortly after the research participants have provided the quantitative data.
The Method and an Illustrative Example
The method for data integration we introduce in this paper consists of seven phases: data
collection, formation of focus groups, data analysis (quantitative), data visualization
(quantitative), data collection (qualitative), data analysis (qualitative), and critical interpretation.
Table 4 introduces the phases of our method, along with detailed information on associated
procedures and products. We have compiled this table in accordance with guidance for visual
modeling of mixed-methods design procedures provided in Ivankova et al. (2006, p. 16). The
mixed methods design used is explanatory sequential (as defined by Creswell et al. 2011)
because it is intended to explain initial quantitative results using a qualitative follow-up
component.
------------------------------------
Insert Table 4 about here
------------------------------------
In order to showcase application of our method, we conducted a study in which we
explored sensemaking of stakeholder groups by managers. The purpose of our study was to
understand the reasoning managers apply when grouping stakeholders and when designing
REAL-TIME INTEGRATION 12
strategies for dealing with each group of stakeholders. Our research question was the following:
How do managers make sense of stakeholder groups?
Sensemaking has been defined as “the ongoing retrospective development of plausible
images that rationalize what people are doing” (Weick et al., 2005, p. 409). This is an important
yet under-researched aspect of stakeholder management (Davis, 2014, p. 192; Turner, 2014;
Turner & Zolin, 2012; Turner & Müller, 2006). The participants’ task in our study consisted of
identifying groups among Fédération Internationale de Football Association’s (FIFA)
stakeholders. FIFA is a suitable case for exploring managers’ sensemaking of stakeholder groups;
it is beholden to a number of stakeholders who require different information about its
performance (Schenk, 2011) and demand different strategic approaches. As revealed by recent
scandals documented in the press (Poddar, 2014), FIFA seems to eschew formal mechanisms of
accountability to its stakeholders (Pielke, 2013). Making the organization more accountable
would require, among other things, making sense of FIFA’s stakeholder groups and designing a
strategy for dealing with each of them.
Our study participants were 50 managers enrolled in a part-time executive MBA program
in Switzerland. Our sample was appropriate for uncovering how managers make sense of
stakeholder groups. The participants were collectively experienced managers (with an average
work experience of 8 years), cross-functional (coming from different functional areas), and
international (coming from 17 different countries). No prior knowledge of FIFA was necessary.
All participants were informed that their participation was voluntary, anonymous, and would not
be graded.
REAL-TIME INTEGRATION 13
Phase 1: Data Collection
In this study, we collected quantitative data through electronic card sorting. To convey the
card sorting instructions, we created a website
(https://sites.google.com/site/cardsortingstakeholders) in which we uploaded information about
FIFA and explained the task that participants were invited to complete. Each participant was
given 10 minutes to read the background information
(https://drive.google.com/file/d/1L1crREVbzFZfmIotZA8-84K-SLZRfR5c/view) and the task
instructions (https://drive.google.com/file/d/1zaVF2PxxEkfwXnATGb8VcInn_VpWEaIh/view).
The background information covered essentials about FIFA’s stakeholders and the problems
associated with managing them. The individual task that each participant had to complete was
formulated as follows:
TASK: “You are a strategy consultant working for FIFA’s president. The president is
facing the challenges of stakeholder diversity and stakeholder pressure. Your task is to give him
an overview of his stakeholder groups in order to help him better understand them. You will have
to sort the stakeholder cards online
(https://www.simplecardsort.com/sort.php?s=JLX122XD&p=4541). Each card will have the
name of a FIFA stakeholder written on it. We would like you to sort the cards into groups. You
are welcome to use any criteria you like and any group labels and subgroup labels you like,
including ‘don’t know,’ ‘not sure,’ and ‘not applicable.’ You can create subgroups (and nest them
into each other). Please read the background information about FIFA and its stakeholders before
you start sorting. You can refer to this information at any time during the sorting task (you do not
have to memorize any specific information).
REAL-TIME INTEGRATION 14
Each participant was then given 20 minutes to complete the individual electronic card
sort for this first phase of our study (as outlined in Table 4). We designed the electronic card
sorting task with the help of the simplecardsort.com online platform. The task was devised as an
“open card sort,” following the procedure documented in Rugg and McGeorge (2005). We
provided no pre-supplied stakeholder groups, and thus aimed “to elicit criteria and categories”
from the participants (Ibid., p. 97). We used the 20 most significant stakeholders of FIFA, as
listed at fifa.com. Figure 1 contains a list of the 20 cards, each bearing the name of one of FIFA’s
20 stakeholders. Each of these cards had to be dragged to the right and grouped with other cards.
-----------------------------------
Insert Figure 1 about here
-----------------------------------
All participants were assembled in a big plenary room and accessed simplecardsort.com
through their laptops or mobile phones. The participants finished their card sorting task within 20
minutes. 50 individual card sorts were stored online at the end of the task. Figure 2 shows an
example of an individual card sort. Each individual card sort shows how each manager
distinguishes and labels FIFA’s stakeholder groups.
-----------------------------------
Insert Figure 2 about here
-----------------------------------
Phase 2: Formation of Focus Groups
We assigned all participants to 6 focus groups (3 groups of 9 people and 3 of 8). Eight to
ten people is the optimal group size recommended by focus group researchers (Krueger & Casey,
2014). Each participant was randomly given a focus group number (from 1 to 6) written on a
piece of paper together with the number for the room to which they should return after a break.
We explained that participation in the focus groups was designed to help us understand the card
sorting data. All participants then left the big plenary room for a break. Random allocation of
REAL-TIME INTEGRATION 15
focus group participants was designed to ensure a non-biased interpretation of the aggregate
quantitative results.
Phase 3: Data Analysis (quantitative)
Simplecardsort.com produced and stored numeric data that contained all individual card
sorts. We then ran a cluster analysis using all these card sorts through the simplecardsort.com
platform. The cluster analysis files, available for download here
(https://drive.google.com/file/d/1jOAyaRLefYLU2TZYHk1ZP3Dwi-
cSWxm1/view?usp=sharing), include a card summary, a group summary, a groups-by-card
summary, a maximum group agreement solution, a participant summary, participant-card raw
data, and a similarity matrix.
Phase 4: Data Visualization (quantitative)
Simplecardsort.com was used to produce a dendrogram representing the aggregate cluster
analysis results. A dendrogram is a tree diagram used to illustrate arrangement of the clusters
produced by hierarchical clustering. The percentages of agreement for each cluster popped up
when the mouse rolled over a cluster in this electronic dendrogram (Figure 3).
-----------------------------------
Insert Figure 3 about here
-----------------------------------
Phase 5: Data Collection (qualitative)
The participants entered the 6 separate focus group rooms after a 20-minute break. The
same interactive dendrogram (see Figure 3) was shown on a large screen in each focus group
room. We facilitated the focus groups by asking questions aimed at interpreting the cluster
analysis results displayed in the dendrogram. The same questions were used in all groups. Table
5 shows the questions and examples of visual outputs (i.e., how the dendrogram was annotated
based on the answers to the questions).
REAL-TIME INTEGRATION 16
-----------------------------------
Insert Table 5 about here
-----------------------------------
The participants answered our questions, but they were also free to add further and
unprompted interpretations. We annotated the dendrogram according to participant input while
the group conversation unfolded. The group decided what was to be annotated and how. The
annotated dendrogram in Table 6 represents a final version of the dendrogram with the
annotations produced in one of the focus groups.
-----------------------------------
Insert Table 6 about here
-----------------------------------
The annotated dendrogram shows that the focus group agreed on the following labels for
the clusters of the dendrogram – “confederations,” “opinion leaders,” “active participants,”
“FIFAs responses to public pressure”, etc. We compared these with the labels provided in the
individual card sorts and found that the focus-group labels were more detailed and exhaustive. In
25 percent of the cases, the individual participants had labeled their groups of cards either
“internal stakeholder”, “external stakeholder”, or a variation containing the word “internal” or
“external” (see “group summary” folder in the cluster analysis files; see also “max group
agreement solution” in the cluster analysis files, where the “internal stakeholders” group label
received the highest agreement).
Once the clusters were labeled, the focus group participants were asked to suggest
strategic measures that FIFA should undertake to manage each labeled cluster (i.e., group of
stakeholders). These suggested strategic measures were also added to the dendrogram. We
obtained two products from our focus groups: 1) six annotated versions of the (same)
dendrogram (archived as jpg. files) and 2) six audio recordings of the focus group conversations.
REAL-TIME INTEGRATION 17
Phase 6: Data Analysis (qualitative)
We transcribed and thematically coded the audio recordings from the focus groups
(following the procedure described in Gläser and Laudel, 2010 and Krueger and Casey, 2014).
We performed three coding cycles – we used peer debriefings to discuss codes and assess if the
conclusions that were reached were plausible. We re-formulated our codes during the process.
The following themes emerged as main topics: “seeking agreement (collaboration) with media
partners”, “peer collaboration among stakeholders”, “non-traditional media”, and “involvement”
(see Table 6 and Meta-inference 2 below). The transcripts of the audio recordings also showed
that the participants often changed their minds about what is worth labeling and how the
stakeholders should be grouped and labeled accordingly. For example, a fifty-sixty percent
agreement threshold was agreed upon in the beginning of most focus group conversations, but
modified later on (see Meta-inference 1 below).
Phase 7: Critical Interpretation
We compared the dendrogram displaying the results of the cluster analysis (quantitative)
with the transcripts of the focus group conversations as well as the textual annotations on the
dendrogram (qualitative). We depicted this comparison on a joint visual display, an excerpt of
which is presented in Table 6. At the top of Table 6 is an example of a dendrogram with the
annotations added during a focus group conversation. The right-hand column in Table 6 contains
verbatim quotations from the transcript of this focus group conversation. These quotations reveal
the reasoning behind the strategic measures suggested for each stakeholder group. For example,
according to the thinking behind the measures suggested for “public independent organizations,”
traditional means of communication do not work because FIFA is not formally accountable to
this stakeholder group. Non-traditional channels should therefore be used in conveying new
REAL-TIME INTEGRATION 18
values, pro-actively providing information about progress made by FIFA in reform, and
establishing partnerships. Accordingly, the following strategic measures were suggested:
“establishing partnerships (dedicated staff), providing pro-active information about reform, and
communicating new values.”
Comparing the coded transcripts with the annotations on the dendrogram, as well as with
the dendrogram itself (which was an aggregate visual representation of the quantitative results),
allowed us to derive the following meta-inferences, which provided answers to our research
question, i.e., “how do managers make sense of stakeholder groups?”.
Meta-inference 1: In making sense of stakeholder groups, managers dynamically redefine
the boundaries of those groups.
As can be seen from the annotated dendrogram (Table 6), the orange lines (added during
the focus group conversation) set labeling thresholds that do not follow the dendrogram’s
clusters blindly. One example is UEFA, which is included under the general “confederations”
label, although UEFA is displayed in a separate cluster on the dendrogram (a cluster with 57
percent agreement). This was explained by our focus group participants by the necessity to treat
all confederations equally, although some, like UEFA, exhibited unique courses of action,
including engagement in public campaigns against FIFAs president. This challenged the
quantitative results.
It is also apparent from the dendrogram that the orange lines do not always follow the
fifty-sixty percent agreement threshold that were tentatively agreed upon in the beginning of the
focus group conversation (see Table 6). For example, politicians were included under a separate
“opinion leaders” label, although the level of agreement corresponding to this dendrogram
cluster was only twenty to thirty percent. This was explained as follows – politicians are a special
REAL-TIME INTEGRATION 19
group that has to be approached separately, as a group of potential opinion “champions” (see
annotated dendrogram in Table 6). “Fans” were also labeled as a separate group. This also
challenged the quantitative results – i.e., approximately 50 percent of all individual card sorts (as
depicted by the respective cluster in the dendrogram) agreed that fans should be placed in the
same group as the unions, associations, clubs, and leagues. Our focus group participants
disagreed, arguing that “the fans are, in fact, a unique group, which deserves unique treatment,
and should be strategically approached through an online forum created especially for them.
Meta-inference 2: Managers make sense of stakeholder groups by primarily trying to
figure out which level of involvement is appropriate for each stakeholder group.
Involvement was recommended as an essential strategic measure (“Involvement: that is
the core. Involve them, make them partners ...” and “Involvement, that is the most important”).
Involvement was intended in a broad sense – from consultation to taking part in FIFAs decision-
making. Although involvement was recommended for most stakeholder groups (e.g., so that
…they cannot say that a decision was wrong, because they were part of it”), our participants
were notably careful about which stakeholder groups should actually be involved. For example,
they did not recommend active involvement of fans. Had the quantitative results (i.e., the
dendrogram) been merely replicated, the fans would probably have received the same label and
consequently the same recommendation for active involvement as the “active participants.” As
indicated by a high-voltage sign in the annotated dendrogram in Table 6, participants perceived
conflict between the “opinion leaders” and the “media and sponsors”. Hence active pursuit of
consensus between the latter two stakeholder groups would need to be encouraged by organizing
peer events. The media representatives could be involved only on the condition that they had
settled their conflict with the politicians beforehand. This revelation would not have been
REAL-TIME INTEGRATION 20
possible based solely on the quantitative results, i.e., the qualitative input extended the
quantitative results.
In a third case, in which involvement was not recommended for the stakeholder group
labeled “FIFAs responses to public pressure,” the qualitative input explained the quantitative
results. Had we relied solely on the quantitative results, it would have remained unclear why
FIFAs internal committees had been clustered in one group together with the medical assessment
center. The “internal committees” label, which had been produced quantitatively, would not have
helped in clarifying the grouping for these two stakeholders. The label produced during the focus
group conversation, namely “FIFAs responses to public pressure,” was more effective in
explaining the recommendation for controlling these two stakeholders groups. The following
explanation was provided by our participants: stakeholders like FIFA’s internal committees,
which had been created solely in response to public pressure, should be controlled because of
their damaged reputation and their historically-evidenced inability to act credibly.
Discussion
Our method produced meta-inferences challenging and extending the quantitative results.
The dendrogram annotations challenged the quantitative clustering. As the focus group
conversations unfolded, participants shifted the threshold lines for some stakeholder groups and
contradicted the dendrogram by dynamically redefining the boundaries of those groups. Neither
was labeling in complete agreement with the dendrogram. Compared with the quantitative labels,
the cluster labels that were added to the dendrogram during the focus group conversations were
more detailed and exhaustive.
Exploring disagreement with the quantitative results, therefore, led to expanded
understanding. According to Fetters et al. (2013, p. 2143), “expansion occurs when the findings
REAL-TIME INTEGRATION 21
from the two sources of data diverge and expand insights of the phenomenon of interest... For
example, quantitative data may speak to the strength of associations while qualitative data may
speak to the nature of those associations.” In our study, the cluster analysis results spoke to the
strength of association with the clusters of the dendrogram. Conversely, the qualitative focus-
group interpretations spoke to the nature of those associations. The final product of the focus
groups – the interpretative annotations added to the Table 6 dendrogram (as labels or “strategic
measure” proposals) – were an integrative reflection of quantitative explicitness merged with
nuances of qualitative thoughtfulness.
Contribution to the Field of Mixed Methods
A widely discussed challenge to mixed methods is the need to make the process of data
integration legitimate in the sense of being meaningful and seamless (Bustamante, 2019;
Guetterman et al., 2015; Ivankova, 2013; Leech et al., 2010; Wall et al., 2013). The unique
contribution of this article to the field of mixed methods research is an approach that facilitates
participant-driven, visually supported, and real-time data integration.
First, the method employs the potential of data visualization to facilitate participant-
driven data integration. Using interactive and annotatable visual representations, researchers and
participants work together to integrate quantitative and qualitative data. Spontaneous
interpretations of results by the participants – in the form of qualitative responses – are inserted
as annotations in the quantitative data visualizations. Hence integration of quantitative and
qualitative data becomes seamless and authentic. The focus is placed “on involving the voices of
the targeted population in the research” (Fetters et al., 2013, p. 2139).
Second, the method facilitates visually supported data integration. The interactive data
visualizations are diagrams (see Table 3), i.e., computer-generated drawings that display
REAL-TIME INTEGRATION 22
information about the geometric and topological relations among the components of the research
problem and express it explicitly. For example, the geometric and topological relations among
the clusters in our study explicitly represented the aggregate results from the card sorting task.
By so doing, these diagrams serve as a guide (Gibson, 1978; Silver, 2008; Suthers &
Hundhausen, 2003) during the follow-up group conversations (in our study, the percentages of
agreement that popped up when the mouse rolled over a cluster within the dendrogram guided
sensemaking).
Third, the method facilitates real-time data integration. “Rapid research feedback”
(Wenger-Trayner et al., 2017, p. 13) is obtained from research participants by conducting focus
groups shortly after the initial quantitative phase. Our research participants join the focus group
conversations with fresh memories of the quantitative phase. By so doing, they are able to
provide credible and meaningful interpretations of the visualized quantitative results. “Time-to-
audience” (Dyllick & Tomczak, 2009, p. 7) of the quantitative research results is reduced from
months (or years) to twenty minutes.
Scope of Application
The method has a broad scope of application. It is useful whenever initial quantitative
results need to be explained or enriched by using a qualitative follow-up component (e.g., in an
explanatory sequential mixed methods design). The latter can be applied provided that the initial
quantitative results can be aggregated in a visual representation (like a bar chart, a dendrogram,
or another computer-supported visual representation). Examples include results of surveys,
experiments, and card sorting, with the latter having been illustrated in our study.
The application of this method is especially suitable for situations in which participant-
driven, visually supported, and real-time data integration is needed. For example, participant-
REAL-TIME INTEGRATION 23
driven integration is needed in the field of management, where the voices of the targeted
population have to be heard to understand the context. The voices of the targeted population may
help researchers understand constituencies affected by wicked problems (see Mertens, 2015).
When practitioners become involved in interpretation of research results, they are better able to
base their subsequent practice on those results – so that mixed methods research can help in
“building a better world” (Molina-Azorin & Fetters, 2019).
Visually supported integration is particularly helpful within heterogeneous or multi-
disciplinary groups or teams. In this case the fact that “different constituencies and stakeholders
do not all value the same kind of information” (Molina-Azorin & Fetters, 2019, p. 280) becomes
especially relevant. Here, the universal nature of the visual language can mitigate linguistic,
disciplinary, or knowledge barriers and provide integrated mixed methods findings that are
compelling for all stakeholders. Real-time data integration would be useful in any managerial or
research situation in which time is a factor.
Limitations
To ensure a broad scope of application for our method, a few basic considerations should
be taken into account in designing the quantitative visual representation (Bresciani & Eppler,
2015). Generally, a visual representation depicts information less precisely than a number or a
table (Few, 2006; Kosslyn, 2006). Researchers need to apply a visual representation which is
adequate for the information to be represented. Some visual representations based on predefined
forms or (technology-driven) templates do not meet this standard (Few, 2006; Tufte, 2001). For
example, if a visual representation is designed to place a focus on some items (Lurie & Mason,
2007), this might channel thinking in a set direction (Mengis, 2007). The large screens needed to
display the visual representation (e.g., the dendrogram) in the center of each focus group room
REAL-TIME INTEGRATION 24
for our study are another limitation on our method. If large screens prove unfeasible, white walls
for data projection or smaller (e.g., laptop or tablet) shared screens can be used instead. In any
case, screens shared via the Internet will be necessary in remote settings. A further limitation of
our method is connected with the risks of ineffective focus group facilitation. To allow
participants to express their genuine thoughts and feelings associated with the quantitative results
displayed in a visual representation, facilitators must refrain from “explaining” the
representation.
Conclusion
In this paper we discussed the theoretical underpinnings and proposed application of a
method in which instant data visualization facilitates real-time data integration and involves
participants in data interpretation. We have proposed an application of this method within the
context of mixed method research, whenever initial quantitative results need to be interpreted,
explained or enriched by using a qualitative follow-up component – quantitative data can be
comprehended, questioned, modified and augmented by research participants. We illustrated our
arguments with an examination of an example study (in an explanatory sequential mixed
methods design), hence providing a discussion of the advantages of our method.
According to Ketokivi and Mantere (2010) reasoning is “incomplete” (p. 315) if it fails to
amplify our knowledge; in other words, the conclusion should be more than a restatement of the
premises. The method introduced in this paper potentially amplifies quantitative knowledge; the
collaborative interpretations of visualized quantitative data offer a “1 + 1 = 3” integration
formula (Fetters & Freshwater, 2015, p. 116). They permit challenging and extending of the
quantitative results in a manner that would not have been possible otherwise. This is a value-
REAL-TIME INTEGRATION 25
adding research outcome of data integration, whereby “analytic density” (Fielding, 2012, p. 127)
is achieved by involving the research participants as co-interpreters.
The implications of the method include its possible application whenever (a) the voices
of the targeted population have to be heard (participant-driven integration), (b) linguistic or
knowledge barriers have to be mitigated (visually supported integration), (c) time is a factor
(real-time integration). The method has the potential to help practitioners better understand
research results, become involved in a real-time interpretation of the collected data, and base
their practices on those results – so that research can have impact in the real world.
REAL-TIME INTEGRATION 26
References
Alexander, E., Bresciani, S., & Eppler, M. J. (2015). Understanding the impact of visual
representation restrictiveness on experience sharing: An experimental assessment. Journal
of Visual Languages & Computing, 31, 30–46.
Alvesson, M., Hardy, C., & Harley, B. (2008). Reflecting on Reflexivity: Reflexive Textual
Practices in Organization and Management Theory. Journal of Management Studies, 45(3),
480–501.
Archibald, M. (2018). Integrating the Arts and Mixed Methods Research: A Review and a Way
Forward. International Journal of Multiple Research Approaches, 10(1), 342–355.
Balomenou, N., & Garrod, B. (2015). A review of participant-generated image methods in the
social sciences. Journal of Mixed Methods Research, (April), 1–17.
Bartunek, J. (2007). Academic-Practitioner Collaboration Need Not Require Joint or Relevant
Research: Toward a Relational Scholarship of Integration. Academy of Management Journal,
50(6), 1323–1333.
Bazeley, P. (2018). Integrating analyses in mixed methods research. London Sage.
Bazeley, P., 2010. Computerized data analysis for mixed methods research. In A. T. & C. Teddlie
(Ed.), Sage handbook of mixed methods in social and behavioral research (2nd ed., pp.
385–422). Thousand Oaks, CA: Sage.
Brechet, C., Baldy, R., & Picard, D. (2009). How does Sam feel?: Children’s labelling and
drawing of basic emotions. British Journal of Developmental Psychology, 27(3), 587–606.
Bresciani, S. & Eppler, M. J. (2015). The Pitfalls of Visual Representations: A Review and
Classification of Common Errors Made While Designing and Interpreting Visualizations.
SAGE Open, pp.1–14.
REAL-TIME INTEGRATION 27
Bryman, A. (2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed
Methods Research, 1(1), 8–22.
Bustamante, C. (2019). TPACK and Teachers of Spanish: Development of a Theory-Based Joint
Display in a Mixed Methods Research Case Study. Journal of Mixed Methods Research,
13(2), 163–178.
Creswell, J., Klassen, A., Plano Clark, V., & Smith, K. (2011). Best practices for mixed methods
research in the health sciences. Bethesda (Maryland): National Institutes of Health.
D’Angelo, A., Ryan, L., & Tubaro, P. (2016). Visualization in Mixed-Methods Research on
Social Networks. Sociological Research Online, 21(2).
Davis, K. (2014). Different stakeholder groups and their perceptions of project success.
International Journal of Project Management, 32(2), 189–201.
Deetz, S. (2008). Engagement as co-generative theorizing. Journal of Applied Communication
Research, 36(3), 289–297.
Dejonckheere, M., Lindquist-Grantz, R., Toraman, S., Haddad, K., & Vaughn, L. (2018).
Intersection of mixed methods and community-based participatory research: A
methodological review. Journal of Mixed Methods Research.
https://doi.org/10.1177/155868981877846
Dyllick, T., & Tomczak, T. (2009). Erkenntnistheoretische Basis der Marketingwissenschaft. In
Qualitative Marktforschung (pp. 65–79). Springer.
Feighery, W. (2006). Reflexivity and tourism research: Telling an (other) story. Current Issues in
Tourism, 9(3), 269.
Fetters, M., Curry, L., & Creswell, J. (2013). Achieving integration in mixed methods designs –
principles and practices. Health Services Research, 48(6 Pt 2), 2134–56.
REAL-TIME INTEGRATION 28
Fetters, M., & Freshwater, D. (2015). Editorial: The 1 + 1 = 3 Integration Challenge. Journal of
Mixed Methods Research, 9(2), 115–117.
Few, S. (2006). Information dashboard design. O’Reilly Sebastopol, CA.
Fielding, N. (2012). Triangulation and Mixed Methods Designs Data Integration With New
Research Technologies. Journal of Mixed Methods Research, 6(2), 124–136.
Flowers, A. A., Carroll, J. P., Green, G. T., & Larson, L. R. (2015). Using art to assess
environmental education outcomes. Environmental Education Research, 21, 846–864.
Garner, J. T. (2015). Different ways to disagree: A study of organizational dissent to explore
connections between mixed methods research and engaged scholarship. Journal of Mixed
Methods Research, 9(2), 178–195.
Gibson, J. (1978). The ecological approach to the visual perception of pictures. Leonardo, 11(3),
227–23.
Guetterman, T., Creswell, J.W. & Kuckartz, U., 2015. Using joint displays and MAXQDA
software to represent the results of mixed methods research. In M. T. McCrudden, G.
Schraw, & C. W. Buckendahl (Eds.), Use of visual displays in research and testing: Coding,
interpreting, and reporting data (pp. 145–175). Information Age Charlotte, NC.
Guetterman, T., Fetters, M., & Creswell, J. (2015). Integrating quantitative and qualitative results
in health science mixed methods research through joint displays. The Annals of Family
Medicine, 13(6), 554–561.
Heron, J., & Reason, P. (2006). The Practice of Co-operative Inquiry: Research with rather than
on people. In P. Reason & H. Bradbury (Eds.), Handbook of Action Research: Concise
Paperback Edition (pp. 144–154). London: Sage.
REAL-TIME INTEGRATION 29
Howe, K., & Eisenhart, M. (1990). Standards for qualitative (and quantitative) research: A
prolegomenon. Educational Researcher, 19(4), 2–9.
Israel, B., Eng, E., Schulz, A., Parker, E. (2013). Methods for community-based participatory
research for health. San Francisco, CA: Jossey-Bass.
Ivankova, N., Creswell, J., & Stick, S. (2006). Using mixed methods sequential explanatory
design: From theory into practice. Field Methods, 18(1), 3–20.
Ivankova, N. V. (2014). Mixed methods applications in action research. Sage.
Ivankova, N. V. (2013). Implementing quality criteria in designing and conducting a sequential
QUAN QUAL mixed methods study of student engagement with learning applied
research methods online. Journal of Mixed Methods Research, 8(1), 25–51.
Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods
research. Journal of Mixed Methods Research, 1(2), 112–133.
Ketokivi, M., & Mantere, S. (2010). Two strategies for inductive reasoning in organizational
research. Academy of Management Review, 35(2), 315–333.
Kieser, A., & Leiner, L. (2009). Why the rigour-relevance gap in management research is
unbridgeable. Journal of Management Studies, 46(3), 516–533.
King, D., & Learmonth, M. (2015). Can critical management studies ever be “practical”? A case
study in engaged scholarship. Human Relations, 68(3), 353–375.
Knight, L., & Pettigrew, A. (2007). Explaining process and performance in the co-production of
knowledge: a comparative analysis of collaborative research projects. In Organization
Studies Third Summer Workshop (pp. 7–9).
Kosslyn, S. M. (2006). Graph design for the eye and mind. OUP USA.
REAL-TIME INTEGRATION 30
Krueger, R. A., & Casey, M. A. (2014). Focus groups: A practical guide for applied research.
Sage publications.
Kuckartz, U. (2018). Personal notes – opinion expressed during MMIRA 2018 workshop,
Wienna.
Larkin, J., & Simon, H. (1987). Why a diagram is (sometimes) worth ten thousand words.
Cognitive Science, 11(1).
Leech, N. L., Dellinger, A. B., Brannagan, K. B., & Tanaka, H. (2010). Evaluating mixed
research studies: A mixed methods approach. Journal of Mixed Methods Research, 4(1), 17–
31.
Lukka, K., & Modell, S. (2010). Validation in interpretive management accounting research.
Accounting, Organizations and Society, 35(4), 462–477.
Lurie, N. H., & Mason, C. H. (2007). Visual representation: Implications for decision making.
Journal of Marketing, 71(1), 160–177.
MacCoun, R. J. (1998). Biases in the interpretation and use of research results. Annual Review of
Psychology, 49(1), 259–287.
Mengis, J. (2007). Integrating knowledge through communication: An analysis of expert-
decision maker interactions (Doctoral thesis). University of Lugano, Switzerland.
Mertens, D. (2015). Mixed Methods and Wicked Problems. Journal of Mixed Methods Research,
9(1), 3–6.
Mohrman, S. A., Gibson, C. B., & Mohrman Jr, A. M. (2001). Doing research that is useful to
practice: a model and empirical exploration. Academy of Management Journal, 44(2), 357–
375.
REAL-TIME INTEGRATION 31
Molina-Azorin, J. F., & Fetters, M. D. (2019). Building a better world through mixed methods
research. SAGE Publications Sage CA: Los Angeles, CA.
Nicca, D., Fierz, K., Happ, M. B., Moody, K., & Spirig, R. (2012). Symptom Management in
HIV/AIDS: A Mixed Methods Approach to Describe Collaboration and Concordance
Between Persons Living With HIV and Their Close Support Persons. Journal of Mixed
Methods Research, 6(3), 217–235.
O’Connell, R. (2013). The use of visual methods with children in a mixed methods study of
family food practices. International Journal of Social Research Methodology, 16(1), 31–46.
Onwuegbuzie, A., & Dickinson, W. (2008). Mixed methods analysis and information
visualization: Graphical display for effective communication of research results. The
Qualitative Report, 13(2), 204–225.
Onwuegbuzie, A. J., & Combs, J. P. (2010). Emergent data analysis techniques in mixed methods
research: A synthesis. In A. T. & C. Teddlie (Ed.), Sage handbook of mixed methods in
social and behavioral research (2nd ed., pp. 397–430). Thousand Oaks, CA: Sage.
Peroff, D. M., Morais, D. B., Seekamp, E., Sills, E., & Wallace, T. (2019). Assessing Residents’
Place Attachment to the Guatemalan Maya Landscape Through Mixed Methods Photo
Elicitation. Journal of Mixed Methods Research.
https://doi.org/10.1177/1558689819845800
Pielke, R. (2013). How can FIFA be held accountable? Sport Management Review, 16(3), 255–
267.
Poddar, A. (2014). FIFA World Cup 2018: Generating Controversies than Excitement, Amity
Research Centers.
Popper, K. (1959). The Logic of Scientific Discovery. Hutchins and Company, London.
REAL-TIME INTEGRATION 32
Presnell, M. (1994). Postmodern ethnography: From representing the other to co-producing a
text. Interpretive Approaches to Interpersonal Communication, 11–43.
Reason, P., & Bradbury, H. (2008). The SAGE Handbook of Action Research: Participative
Inquiry and Practice. SAGE Publications.
Rugg, G., & McGeorge, P. (2005). The sorting techniques: a tutorial paper on card sorts, picture
sorts and item sorts. Expert Systems, 22(3), 94–107.
Rynes, S., Bartunek, J., & Daft, R. (2001). Across the great divide: Knowledge creation and
transfer between practitioners and academics. Academy of Management Journal, 44(2),
340–355.
Schenk, S. (2011). Safe hands: building integrity and transparency at FIFA. Berlin:
Transparency International.
Shannon-Baker, P. (2015). “But I wanted to appear happy”: How using arts-informed and mixed
methods approaches complicate qualitatively driven research on culture shock.
International Journal of Qualitative Methods, 14(2), 34–52.
Shannon-Baker, P., & Edwards, C. (2018). The Affordances and Challenges to Incorporating
Visual Methods in Mixed Methods Research. American Behavioral Scientist.
https://doi.org/10.1177/0002764218772671
Silver, M. S. (2008). On the Design Features of Decision Support Systems: The Role of System
Restrictiveness and Decisional Guidance. In F. Burstein & C. W. Holsapple (Eds.),
Handbook on Decision Support Systems 2: Variations (pp. 261–291). Springer.
Suthers, D., & Hundhausen, C. (2003). An experimental study of the effects of representational
guidance on collaborative learning processes. The Journal of the Learning Sciences, 12(2),
183–218.
REAL-TIME INTEGRATION 33
Tubaro, P., Ryan, L., & D’Angelo, A. (2016). The visual sociogram in qualitative and mixed-
methods research. Sociological Research Online, 21(2), 1.
Tufte, E. R. (2001). The visual display of quantitative information (2nd ed.). Cheshire, CT:
Graphics Press.
Turner, J. R. (2014). Handbook of project-based management (Vol. 92). McGraw-hill New York,
NY.
Turner, J. R., & Müller, R. (2006). Choosing appropriate project managers: Matching their
leadership style to the type of project. Project Management Institute.
Turner, R., & Zolin, R. (2012). Forecasting success on large projects: developing reliable scales
to predict multiple perspectives by multiple stakeholders over multiple time frames. Project
Management Journal, 43(5), 87–99.
Van Aken, J. (2004). Management research based on the paradigm of the design sciences: The
quest for field- tested and grounded technological rules. Journal of Management Studies, 41,
219–246.
Van de Ven, A. (2007). Engaged Scholarship: A Guide for Organizational and Social Research.
OUP Oxford.
Wall, K., Higgins, S., Remedios, R., Rafferty, V., & Tiplady, L. (2013). Comparing analysis
frames for visual data sets using pupil views templates to explore perspectives of learning.
Journal of Mixed Methods Research, 7(1), 22–42.
Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (2005). Organizing and the process of
sensemaking. Organization Science, 16(4), 409–421.
REAL-TIME INTEGRATION 34
Wenger-Trayner, B., Wenger-Trayner, E., Cameron, J., Eryigit-Madzwamuse, S., & Hart, A.
(2017). Boundaries and boundary objects: an evaluation framework for mixed methods
research. Journal of Mixed Methods Research. https://doi.org/10.1177/1558689817732225
REAL-TIME INTEGRATION 35
Tables and Figures
Table 1
An Example Visual Representation Used in Mixed Methods Research Corresponding to Tufte’s
Third Level
Visual representation
Description
Level
text
-
– summarize numeric data by type (i.e., demographics, source, time, group
membership, scale, level) by “arrang[ing] the type to facilitate comparison” (Tufte, 2001, p.
178)
level 3
(Tufte, 2001)
Purpose: data integration/interpretation, data communication
Example: This table summarizes and arranges numeric data by a) “symptoms” according to a questionnaire, b)
“appraisal of symptoms” according to contrasting groups of participants in narratives elicited after the
questionnaire, and c) the results of testing hypotheses regarding differences in the participants’ narratives.
Source: Nicca et al. (2012, p. 229), re-printed with permission from the authors and the publisher
REAL-TIME INTEGRATION 36
Table 2
An Example Visual Representation Used in Mixed Methods Research Corresponding to Tufte’s
Fourth Level
Visual representation
Description
Level
supertables
– “elaborate tables” which provide “organized, sequential detail, and
reference-like quality” (Tufte, 2001, p. 178) and may contain pictures
level 4
(Tufte, 2001)
Purpose: data integration/interpretation
Example: This elaborate table, a supertable, contains numeric, textual, and visual data (the experimental conditions
are visually represented). The supertable allows comparisons between the shared and withdrawn thoughts
regarding project experiences. The horizontal rules divide the data into group-related paragraphs; the rows are
ordered so as to tell an ordered story about the experimental groups and the individual members of the
experimental groups (based on integrated thoughts).
Source: the authors – table used at the data interpretation stage for Alexander et al. (2015)
REAL-TIME INTEGRATION 37
Table 3
Example Visual Representations Used in Mixed Methods Research Corresponding to Tufte’s Fifth
Level
Visual representation
Description
Level
graphics
(e.g.,
diagrams
) – “combine words, numbers” (Tufte, 2001, p. 178), show
arrangement and relations, and “preserve explicitly the information about the topological
and geometric relations among the components of the problem” (Larkin & Simon, 1987, p.
66)
level 5
(Tufte, 2001)
Purpose: data integration, data communication
Example: This diagram was used to represent the integration of quantitative and qualitative data: black represents
quantitative data, white qualitative data, and gray represents the mixing of black and white – in this case, the
outcome, or “fit”, resulting from integrating the quantitative and qualitative data.
Source: Bustamante (2019, p. 171), re-printed with permission from the author and the publisher
Purpose: data elicitation
Example: This diagram (a sociogram) depicts the personal social network of a research participant. The concentric
circles represent relational proximity to ego, while the quadrants represent the context of relationships in the social
network of each research participant. Research participants were asked to fill in the blanks between the circles.
REAL-TIME INTEGRATION 38
Source: Tubaro et al.3 (2016, p. 7), re-printed with permission from the author Louise Ryan and the publisher
REAL-TIME INTEGRATION 39
Table 4
An Overview of the Method and its Application in our Study (italic)
REAL-TIME INTEGRATION 40
REAL-TIME INTEGRATION 41
Research Phase
Procedure
Product
Du-
ration
(time)
1..Data collection
(quantitative)
Each research participant individually provides data
by completing an individual task, such as an
electronic survey or card sorting.
● numeric data stored
electronically (online)
30
min
in our
study
electronic card sorting
Each research participant completed an individual
electronic card sorting session online. Number of
participants: 50
● numeric data stored
electronically (online, at
simplecardsort.com), containing
all individual card sorts
2..Forming focus groups
The research participants are randomly or
systematically assigned to focus groups.
Focus groups of 8-10 people
2
min
or
longer
in our
study
All 50 participants were assigned to focus groups. 6 focus groups
3..Data analysis
(quantitative)
The researchers run the numeric data through an
analysis tool.
● electronic files containing the
descriptive and inferential
results from the quantitative
phase
15
min
in our
study
We ran the numeric data through
simplecardsort.com. Simplecardsort.com performed
the cluster analysis.
● electronic files containing the
cluster analysis results from the
card sorting
4..Data visualization
(quantitative)
The analysis tool is used to produce a visualization
of the aggregate quantitative results (e.g.,
dendrograms, bar charts).
● electronic (interactive and
annotatable) visual
representation
5
min
in our
study
Simplecardsort.com was used to produce an
interactive dendrogram, which represented the
aggregate cluster analysis results.
● an electronic (interactive and
annotatable) dendrogram
representing the aggregate
cluster analysis result
5.
.
Data collection
(qualitative):
integration through instant merging
The electronic visualization is shown to research
participants in focus groups. The researchers
facilitate focus group conversations aimed at
interpreting the quantitative results displayed in the
visual representation. The researchers ask the
participants to explain the results. The researchers
annotate the visual representation according to
participants input.
audio recordings of the focus
group conversations, which
contain data interpretations
extending (or challenging) the
quantitative results
● annotated visual representation
45
min
in our
study
The electronic visual representation was a
dendrogram.
● audio recordings and
annotated dendrogram (the
same dendrogram with multiple
annotations)
6..Data analysis
(qualitative)
The audio recordings are transcribed and coded. coded transcripts of audio
recordings and identification of
key themes
5
days
in our
study
The audio recordings were transcribed and coded. coded transcripts of audio
recordings and identification of
key strategic suggestions
7..Critical interpretation
of quantitative and qualitative results
(data integration through a joint
display)
The researchers interpret the qualitative and
quantitative findings by merging: the annotated
visual representation is compared with the coded
transcripts of the audio recordings.
a joint display 10
days
in our
study
We compared the annotated dendrograms
with the
coded transcripts of the audio recordings. We
merged the annotations and excerpts from the
coded transcripts on a joint display.
a joint display, which contains
insights further extending (or
challenging) the quantitative
results
REAL-TIME INTEGRATION 42
Table 5
Questions Asked, Examples of Answers, and Visual Outputs
Questions (asked by the facilitator in each focus
group) & Example Answers (by focus group
participants)
Visual outputs
(how the dendrogram was annotated based on the answers to the questions)
Question: Facilitator: In your opinion,
which clusters of the dendrogram are worth
labeling? Please suggest where to draw the
lines of what is worth labeling.
Answers: Participant F: I think … around
sixty percent.
Participant G: OK, this makes sense, let’s
label all around … fifty percent, and
neglect all of the ones that are below this
line…
Facilitator: Does everyone agree with fifty-
sixty percent?
Participant G: Yes, it is a good idea, so that
we know what to label.
Participant F: Let us neglect the ones with a
small percentage. Let’s concentrate on the
middle ones, because they are the most…
Let’s draw a line there, hmm…, at around
fifty-sixty percent agreement.
Participant B: Yes, this makes sense.
Facilitator: Ok, let us do this. [Facilitator
starts drawing lines on the dendrogram.]
Threshold lines (in orange) were added to indicate what was worth
labeling:
Question: Facilitator: What labels would
you give to the clusters of the dendrogram?
Please suggest the best possible labels.
Answers: Participant G: The thing is, they
aren’t running to any clubs, or any
confederations – they are general.
Participant F: Independent?
Participant G: Yeah, …
Participant F: No…hmm…
[Participant C points toward Participant F
and helps him find the right word].
Participant C: Public.
Participant F: Yes, exactly, they are public.
Participant A: How about the internals – ex
controllers?
Participant F: The medical assessment
centre was created thanks to public
pressure.
Participant C: Yea, so the public intervened
to regulate an issue.
Facilitator: What’s a good label here?
Participant F: Public influencers? Labels were added to the clusters of the dendrogram:
REAL-TIME INTEGRATION 43
Participant A: FIFA founded the Medical
Assessment Center just in order to avoid
the risk of being excluded from the
Olympic Games.
Participant F: So, responses to public
pressure?
Facilitator: OK, responses to public
pressure [Facilitator writes this label on the
dendrogram.] Does everyone agree that this
is a good label?
Question: Facilitator: What strategic
measures should FIFA undertake to
manage each labeled cluster (of
stakeholders)?
Answers: Participant B: Of course, they
need to participate.
Participant C: With “Play the Game”?
Facilitator: And who else?
Participant B: “Transparency International”
– would you make your opponent part of
your decision process?
Participant C: You can question them,
but…
Participant A: No, no…
Participant C: They cannot say that a
decision was wrong, because they were part
of it.
Participant B: You can establish
partnerships with them.
Participant A: Proactively…
Suggested strategic measures were added to the dendrogram (on the
right-hand side):
REAL-TIME INTEGRATION 44
Table 6
A Joint Display of Dendrogram Annotations from the Focus Groups and Verbatim Quotations
from the Transcripts of the Focus Group Conversations
Annotated dendrogram (example from one focus group)
Suggested measures
(for… stakeholder group)
Quotes from the transcripts of the focus group conversations
communicate, inform, involve,
collaborate (for
“Confederations” – see
annotated dendrogram above)
TOPIC: Seeking agreement (collaboration) with media partners
Facilitator: What strategic measures should FIFA undertake to manage this cluster of
stakeholders? …
Participant: Involvement, that is the most important.
Participant: … and collaborating.
Participant: Collaborating meaning following their rules, or... playing their game?
Participant: No, I think, helping them grow their regions, supporting them in reaching
agreement with their regional media partners, … collaborating in reforming FIFA
governance.
TOPIC: Peer collaboration among stakeholders
enlist as champions, involve
more actively support peer
actions, hold forums (for
“Opinion Leaders” – see
dendrogram)
Facilitator: … Any other ideas regarding opinion leaders or politicians?
Participant: Probably, to organize more peer actions involving those people. Like
forums, or invite them to give speeches.
Participant: Keep them closer, together.
consult, monitor, organize info
events (for “Active
Participants” – see
dendrogram)
Participant: Involvement: that is the core. Involve them, make them partners, in some
committees.
Participant: But do you mean “involvement” as to make decisions? Bypassing the
confederations, that’s a bit tricky. You don’t want to alienate your owners.
Participant: It’s probably more partnership in actions. …
Participant: Organizing events where they get to know each other. Because unions
and clubs aren’t communicating with each other, they may communicate with FIFA,
REAL-TIME INTEGRATION 45
but…
Participant: Yeah, and they never contact FIFA, they hear the negative things only
through the press. They may want to get a little bit closer.
establish partnerships
(dedicated staff), provide pro-
active information about
reform progress, communicate
new values (for “Public
Independent Organizations” –
see dendrogram)
Participant: I would do something else with the independent organizations as well.
Facilitator: What should I write?
Participant: Be more pro-active.
Participant: I would even say that they need to develop their governance together.
Participant: Yeah.
Facilitator: Oh, really?
Participant: Yeah.
Participant: I don’t know. I know they are important, but I think it would be too much
for FIFA to involve all these external organizations, and, moreover, to try to get
consensus between all of them to make a decision in FIFA. So, then you would have
multiple opinions on issues, on where you want to go, and – never a decision, so…
Participant: Yeah, if you engage stakeholders, they might contradict themselves. So,
let’s delete “involve”.
Participant: But if they would come together as a group with a recommendation for
FIFA, that would be different.
Participant: Yes, but you don’t want to really support them, so that they can speak in
one voice against you?
Participant: The thing is, how would the Olympic Committee agree with
Transparency International? I mean, at Transparency International we even have
concerns regarding the Olympic Committee, so?
establish partnerships
(dedicated staff), pro-active
information about reform
progress, communicate new
values (for “Public
Independent Organizations” –
see dendrogram)
TOPIC: Non-traditional media
Participant: If FIFA is to be held accountable, as most of its stakeholders demand,
this can hardly be achieved by traditional means. The reasons for this are two-fold:
first, unlike a public company, FIFA is not answerable to stakeholders. So, do we
have to consider this point? They are not answerable, so is this mandatory?
Facilitator: Legally, they are not answerable. But, based on the pressure they are
getting – they are.
Participant: They can’t do everything for everyone. They have to segment their
actions.
online forum (for “Fans” – see
dendrogram)
Facilitator: Moving right along to the fans… What’s the right way to deal with them?
What’s FIFA’s role in managing this stakeholder group? What would you
recommend?
Participant: An online forum – for sure. It would be really appreciated by the fans if
they would count on their opinion.
Participant: Yeah, and FIFA can be more active in participating in a dialogue.
Facilitator: A very good suggestion. Does everybody agree?
Participant: Yeah.
REAL-TIME INTEGRATION 46
Figure 1
The Individual Card Sorting Workplace
REAL-TIME INTEGRATION 47
Figure 2
Example of an Individual Card Sort
REAL-TIME INTEGRATION 48
Figure 3
An Interactive Electronic Dendrogram Representing the Aggregate Cluster Analysis Results for
all 50 Individual Card Sorts. (The percentage represents the level of agreement for a particular
cluster).
Endnotes
1. The simplecardsort.com online shut its online services down on October 1st, 2019.
Other online card sorting tools include OptimalSort, Proven by Users Online Card
Sorting, UserZoom, uzCardSort, xSort, UsabiliTest Card-Sorting (for a list see
http://www.measuringuserexperience.com/CardSorting/index.htm).
... (p. 4) Y como uno de los principales problemas que se manifiestan en la elaboración de los artículos, está la ineficiente visualización de información y de conocimiento, encargada de facilitar la representación, comprensión y comunicación del creciente volumen de información social proveniente de diversas fuentes y de la gran variedad de conceptos complejos y abstractos que deben procesarse para dar sustento y coherencia a lo que se escribe Díaz, 2018;Cuschieri et al., 2019;Alexander et al., 2020). Siendo la información social el tipo más alto, complejo y multiforme de información, al ser la sociedad la forma más elevada de movimiento de la materia. ...
... Respecto al indicador 1 (Correspondencia entre las formas de presentación de la información y la intencionalidad investigativa declarada por el investigador), en Alexander et al. (2020); Vilaplana (2019) y Bresciani & Eppler (2015) se corroboró que la representación de información constituye una de las tareas del procesamiento de información donde ocurren numerosos errores por parte de investigadores de las ciencias sociales, los que pueden estar asociados a inconvenientes de visualización de naturaleza cognitiva (los más frecuentes), social o emocional. ...
Article
Full-text available
Todo proceso de investigación científica debe hacer galas de una esmerada rigurosidad en la obtención y presentación de sus resultados, para lograr efectividad en su posterior aplicación. El objetivo de este trabajo fue determinar las principales insuficiencias existentes en el procesamiento y comunicación de resultados de investigación, a través de artículos publicados en tres revistas de ciencias sociales de la Universidad de Oriente, Cuba. La investigación tuvo un carácter exploratorio. Se seleccionó una muestra de 187 artículos pertenecientes a los años 2017 al 2019 y se utilizaron métodos cuantitativos para analizarlos. Los resultados indican que existen insuficiencias en el empleo de las TIC para procesar información científica, en la aplicación de métodos y técnicas de investigación que no consiguen captar la complejidad y diversidad de la realidad social, en el procesamiento de grandes volúmenes de datos y en la estética y creatividad con que se presenta dicha información. Estos resultados evidencian la necesidad de una intervención pedagógica, orientada a perfeccionar el desempeño de los profesionales de estas ciencias respecto al procesamiento y comunicación de sus resultados científicos.
... The literature suggests that meaningful integration of qualitative and quantitative data remains elusive because researchers of MMR experience difficulties in integrating the analysis and interpretation of the qualitative and quantitative datasets (Alexander et al., 2021;Lynam et al., 2020). ...
Article
Despite the potential of mixed methods research (MMR) in providing a comprehensive picture of development issues, its pervasiveness and use in development studies is limited. This paper examines the use of MMR as reflected in contributions to the Ghana Journal of Development Studies (GJDS). Based on a rapid review and content analysis of 105 articles, published in the GJDS over the period 2010 to 2017, this paper illustrates there is an inadequate use of MMR among the community of researchers contributing articles to the GJDS. Specifically, only 16 percent of journal articles used MMR, whereas 52 percent and 32 percent used solely quantitative and qualitative approaches, respectively. The huge use of mono-methods and the paucity of MMR in the field of development studies suggest that the 'paradigm wars' and the 'incompatibility thesis' are not over. Eighty-six percent of articles that reportedly used MMR did not mention the purposes for employing it, explain the typologies of its designs used, the stages, or even the way the qualitative and quantitative data were integrated.
... Second, RIRE can help illuminate the step-by-step decision-making process in research generally and in mixedmethod research. The latter being a well-documented challenge (Alexander et al., 2020;Bryman, 2006;Bustamante, 2019;Gutterman et al., 2019;Johnson et al., 2019). RIRE may also be a bridge that connects reflexivity operating unconsciously, the unconscious predilection for making decisions in research, and activating reflexivity to operate in the foreground as a central tool (see Biddle & Schafft, 2015;Cain et al., 2019;Hesse-Biber, 2010). ...
Article
Full-text available
This article highlights “reflexive integration” as a methodological tool that can facilitate the explicit integration of quantitative or qualitative elements in mixed-methods research. Reflexive integration of research elements (RIRE) is advocated as a mechanism that can be used in any mixed-method study to enhance depth of inquiry and transparency of the steps involved in mixed-method research. An illustrative example is presented to show the step-by-step process of reflexive integration at various stages of a mixed-method study.
... They illustrated the integration of the study data and included a joint display of their quantitative and qualitative results side-by-side along with the meta-inferences yielded. Alexander et al. (2020) advanced a visual process for integration that involves research participants. Using this innovative approach, they developed visuals to represent quantitative data collected from participants in a card sort and then used those visuals in a focus group to gather participant feedback and to annotate the visuals. ...
... How could mixed methods social network analysis (Fielding & Cisneros-Puebla, 2009;Yousefi Nooraie et al., 2020), mixed methods GPS (Christensen et al., 2011), or mixed methods GIS (Bhuyan & Zhang, 2019;Jones, 2017) be used to understand and curb the epidemic? How could an approach using machine learning, natural language processing, and data visualization be integrated as mixed methods to develop new understandings (Alexander et al., 2020;O'Halloran et al., 2018)? How could self-reports of COVID-19 symptoms be incorporated into heat maps, analyzed, and utilized real-time to help slow the spread of the virus (Flatten, n.d.)? ...
Article
Full-text available
Joint displays provide a visual means to represent the integration of qualitative and quantitative research in addition to a framework for thinking about integration and organizing data, methods, or results. Despite increases in the use of joint displays, opportunities exist for more creative joint displays that use additional visuals to more easily communicate complex information. These additional visual features include charts, graphs, maps, and images. However, little has been written about their usage within joint displays. The purpose of this methodological article is to advocate the use of joint displays that incorporate graphs, charts, maps images, and other visuals, as appropriate and to discuss the decisions in including these features. To assist in identifying joint displays that include visuals, we conducted a systematic literature search of Google Scholar, PubMed, ERIC, and Academic Search Premier using terms for mixed methods research. After screening articles to identify joint displays that include graphs, charts, maps, images, and other visuals, we analyzed articles (n = 33) for mixed methods features and joint display features. Regarding the quantitative strand in a joint display, charts, and graphs can communicate more information than statistical numbers, such as showing distributions of data, plotting relationships among variables, and using bars of varying lengths to facilitate comparison. Maps and GIS data can similarly relate additional information for the reader, particularly when geographical or spatial area is important to the research. Furthermore, images can be a useful type of qualitative data and is especially relevant in photo-elicitation research. These visuals can be depicted in joint displays to represent integration. Visuals used in joint displays included: column or bar charts, histograms, boxplots, scatter plots, quantitative path models, maps, pictures, and qualitative visual models. We also include four exemplars of joint displays that use visuals. Researchers can use these types of joint displays for integration in psychological intervention research, for theory development in psychology, and for instrument development in educational psychology. We conclude with recommendations for including visuals and suggestions to optimize integration from a mixed methods perspective.
Article
Full-text available
Although mixed methods research (MMR) and community-based participatory research (CBPR) have been employed to investigate complex research questions to improve the reach, rigor, and relevance of research, little is understood about the intersection of the approaches. We conducted a methodological review of studies (n = 129) using both MMR and CBPR, an advanced application we refer to as mixed methods community-based participatory research (MMCBPR). We systematically examined published MMCBPR studies to identify the methodological features and use in current research. Findings demonstrate that the components of MMR were not adequately described although some detail was provided about the use of CBPR. This study contributes to the evolution of advanced applications, and we offer recommendations for future applications of MMCBPR.
Article
We developed mixed methods photo elicitation to mitigate cultural and language barriers and to acquire deeper understandings of indigenous participants’ place attachment. We define mixed methods photo elicitation to integrate quantitative rankings of photos with qualitative induction of the meanings ascribed to the photos. Multidimensional scaling is used to thematically analyze the resulting photo clusters in relation to qualitative investigation of photo meanings. We also introduce a novel approach to a mixed methods joint display, which was used to visualize emerging themes and reveal how quantitative and qualitative findings are integrated. Reacting to a collection of landscape photographs endemic to rural Guatemala, indigenous farmers expressed place dependence to landscapes for economic and noneconomic reasons, and place identity for sociocultural reasons.
Article
Mixed methods researchers are increasingly utilizing visual methods, including portraiture, symbolic drawing, and photo-elicitation. Such methods have been used to capture that which is not observable, communicate experiences that are difficult to verbalize, and promote participants’ self-awareness. Challenges to this approach, however, include limited grounding in the literature, appropriate training, and ethical concerns. After establishing a typology for visual methods, we discuss these affordances and challenges to using this approach in mixed methods studies. We analyze three example studies that utilize different visual approaches to identify their unique and important contributions. We conclude with several key considerations for researchers.