Content uploaded by Manuel Goyanes

Author content

All content in this area was uploaded by Manuel Goyanes on Sep 15, 2020

Content may be subject to copyright.

Content uploaded by Marton Demeter

Author content

All content in this area was uploaded by Marton Demeter on Sep 14, 2020

Content may be subject to copyright.

Vol.:(0123456789)

Scientometrics

https://doi.org/10.1007/s11192-020-03680-6

1 3

A mathematical approach toassess research diversity:

operationalization andapplicability incommunication

sciences, political science, andbeyond

ManuelGoyanes1,2 · MártonDemeter3 · AureaGrané4 ·

IreneAlbarrán‑Lozano4· HomeroGildeZúñiga2,5,6

Received: 28 March 2020

© Akadémiai Kiadó, Budapest, Hungary 2020

Abstract

With today’s research production and global dissemination, there is growing pressure

to assess how academic ﬁelds foster diversity. Based on a mathematical problem/solve

scheme, the aim of this study is twofold. First, the paper elaborates on how research diver-

sity in scientiﬁc ﬁelds can be empirically gauged, proposing six working deﬁnitions. Sec-

ond, drawing on these theoretical explanations, we introduce an original methodological

protocol for research diversity evaluation. Third, the study puts this mathematical model

to an empirical test by comparatively evaluating (1) communication research diversity

in 2017, with respect to ﬁeld’s diversity in 1997, and (2) communication research and

political science diversity in 2017. Our results indicate that, contrasted to pattern diver-

sity, communication research in 2017 is not a diverse ﬁeld. However, throughout the years

(1997–2017), there is a statistically signiﬁcant improvement. Finally, the cross-comparison

examination between political and communication sciences reveals the latter to be signiﬁ-

cantly more diverse.

Keywords Research diversity· Diversity· Communicationscience· Political science·

Diversity gaps

In recent decades, research diversity has become a central element in shaping the form and

content of scientiﬁc ﬁelds (Metz etal. 2016), mirroring the growing societal and economic

demands and pressures of most democratic societies (Dhanani and Jones 2017). With the

growing globalization of academia, diversity enables new opportunities to conﬁgure inclu-

sive scientiﬁc ﬁelds (Waisbord and Mellado 2014; Waisbord 2016), build upon the devel-

opment of plural approaches to scientiﬁc facts and knowledge progress (Stephan and Levin

1991). There is a general consensus that research diversity points to the matureness and

Electronic supplementary material The online version of this article (https ://doi.org/10.1007/s1119

2-020-03680 -6) contains supplementary material, which is available to authorized users.

* Manuel Goyanes

mgoyanes@hum.uc3m.es

Extended author information available on the last page of the article

Scientometrics

1 3

sophistication of most academic disciplines (Wasserman 2018), enriching empirical evi-

dences with plural visions of the world (Livingstone 2007; Willems 2014), and challenging

the taken-for-granted assumptions of academic elites (Demeter 2018). However, despite

the importance of rigorously measuring the state in which diﬀerent intellectual terrains

are positioned regarding research diversity, little research has directly developed a reliable

instrument to both evaluate diversity claims and infer the potential diversity gaps that exist

in the academia. This paper seeks to palliate this gap, proposing a protocol to evaluate

research diversity, from a multivariate perspective, based on six working deﬁnition. We

illustrate this protocol in the ﬁelds of Communication and Political Science.

For the genesis of this article we took the following approach: initially, we conduct a

brief literature review of diversity measures in general, and in bibliometric studies in com-

munication research in particular, with the aim of designing a research diversity frame-

work, conceptualizing the main items and scales often used for gauging research patterns

in the ﬁeld. Despite extant research on communication studies seldom address the potential

formulas to measure diversity in research (an exception would be Leydesdorﬀ and Probst

2009), they provide critical perspectives, variables and measurements to assess the evolu-

tion of the ﬁeld (in terms of authorships, methodologies, thematic approaches) and thus

the potential diversity of its core components (Freelon 2013; Günther and Domahidi 2017;

Walter etal. 2018). After the literature review, we propose, deﬁne and describe a method-

ology and the associated research protocol to calculate the research diversity of a given

ﬁeld and its research production.

Since our interest is in Communication Sciences, we apply these measurements to cal-

ibrate this discipline ﬁrst. Speciﬁcally, we conducted a content analysis of a representa-

tive and randomized sample of articles (N = 283) published in all Journal Citation Reports

(JCR) journals (NJ = 84) indexed under the category of “communication” in 2017. In addi-

tion, we assess the current diversity of research in Communication Sciences compared to

that of 20years ago (N = 263; NJ= 36), following the same methodological procedure out-

lined above. Finally, we compare this research diversity with that of a cousin ﬁeld, i.e.

Political Science (N = 329; NJ = 169). In all cases, sample sizes were calculated with a

conﬁdence level of 95%. Therefore, assuming normality, the ﬁnal samples had a sampling

error of less than 5%.

Measuring diversity: abrief historical overview

Measuring diversity has a long tradition (Rao 1982a). The ﬁrst attempts to provide reli-

able diversity measurements date back as those initial eﬀorts of Gini in economics (Gini

1912), Sokal and Sneath in biology (1963), Agresti and Agresti in sociology (1978) or Rao

in anthropology (1948). Rao (1982a) reviewed some of these measures and oﬀered three

uniﬁed approaches for deriving them (Rao 1982a), providing also diversity decomposition

examples within a population in terms of given or conceptual factors (Rao 1982b). Later

scientometric scholars interested in diversity issues mostly adopt and modify Rao’s indices,

showcasing the strong inﬂuence of Rao’s works (Leydesdorﬀ etal. 2019; Stirling 2007), in

applying diversity measures on diﬀerent levels of analysis, including individual journals

(Zhang etal. 2009, 2010), and articles (Zhang etal. Zhang etal. 2016).

Stirling (2007), who partially built his approach on Rao’s calculations (1982a), con-

sidered diversity as an attribute of all systems whose elements could be appointed into

diﬀerent categories. These three systemic features are: variation, balance and disparity.

Scientometrics

1 3

By reference of ten quality criteria, Stirling proposes a new general diversity heuristic in

which each of the aforementioned three subordinate properties—variation, balance and

disparity—could be systematically explored. Later scholarships typically adopted Stirling’s

insights regarding the use of variation, balance and disparity in gauging diversity (Rafols

and Meyer 2010; Ráfols 2014).

Bone and his colleagues (Bone etal. 2019) deﬁned diversity in line with Stirling’s con-

ceptualization (Stirling 2007), too, but as opposed to Striling (2007) and Ráfols (2014),

they measured distances between individuals, and not categories. By conceptualizing

diversity on this basis, they followed Boschma work (2005) who established the concept

of proximity as a key concept in diversity calibration. Boschma and his later followers

applied ﬁve forms of proximity, namely cognitive, organizational, social, institutional and

geographical proximity, where greater proximity in each category means greater diversity.

More recently, Leydesdorﬀ and Ráfols (2010) analyze diﬀerent indices by which inter-

disciplinarity could be quantitatively measured, such as Gini coeﬃcients, Shannon entropy

indices, and the Rao-Stirling diversity index. Later research showed that using Rao–Stirling

diversity (RS) indices sometimes produces anomalous results (Leydesdorﬀ etal. 2019). It

is typically argued that these anomalies could be related to the use of the dual-concept

diversity that combines both balance and variety (Stirling 2007). Based on this observa-

tion, Leydesdorﬀ et al. (2019) modiﬁed RS into an index that operationalizes the three

diversity features of Stirling—variety, balance and disparity—independently, and then

combines them ex post. This formula has been criticized and slightly modiﬁed later by

Rousseau (2019).

The contribution of our study is as follows: instead of providing a speciﬁc formula or

comparing diﬀerent formulas, we propose an entire protocol to gauge the diversity of a

given academic ﬁeld based on some speciﬁc characteristics of its authors and the type and

features of the research they carry out. While the Stirling–Rao indices (and also Simpson

diversity indices) are measures of the internal diversity of a variable (and the Stirling–Rao

index also incorporates a measure of distance between categories), our proposal is based

on comparisons to a certain “diversity pattern”. For example, in Rafols and Meyer (2010),

diversity formulas are used to compare diﬀerent disciplines through the variable “ref-of-

refs” along with a matrix of dissimilarities between disciplines. On the contrary, our con-

cept of “variable diversity” is deﬁned as a battery of measures that allow us to compare

the variability of each of the variables of interest with its corresponding pattern. We have

illustrated these comparisons using Hellinger’s distance, but any other distance function

between probability distributions might be valid. Finally, we take the average of all dis-

tances as a comprehensive measure of the ﬁeld variability. We remark that the choice of the

distance function is not as important as the calibration of the threshold, from which it will

be decided if the variable of interest follows or not the given diversity pattern. This calibra-

tion is done via bootstrap.

Communication research patterns: literature review

While we still lack a sound deﬁnition for research diversity and a reliable measurement

for its calibration, there is a robust body of literature that, either explicitly or implicitly,

problematizes diversity issues in communication research. In the following subsections, we

present the main empirical contributions of these research branches, explaining how our

Scientometrics

1 3

study contributes to further evaluate diversity claims and infer the position and evolution of

single or multiple ﬁelds of science.

Methodological, disciplinary andtheoretical foundations ofdiversity

incommunication studies

Analyses of publication patterns in communication studies can be found as early as 1989,

when the special issue Communication Research was ﬁrst published on this topic (Vol 16

Issue 5). In the same year, Journal of Communication also dedicated three special issues

to analyzing publication patterns, as well as the most frequently assessed subﬁelds in com-

munication research (Vol 43 Issue 3, Vol 54 Issue 4 and Vol 55 Issue 3), showcasing the

growing relevance of such meta-scholarship to evaluate the state of the ﬁeld. Paradoxically,

the ﬁrst citation analysis of communication journals was also published in Paisley (1989),

followed by a brand-new research stream on bibliometric or scientometric studies. This

study contributes to this research tradition by assessing the empirical, methodological and

thematic evolution of the discipline (Funkhouser 1996; Reeves and Borgman 1983; Rice

et al. 1988; Borgman 1989; Rogers 1999; Feeley 2008; Bunz 2005; Griﬃn etal. 2016;

Keating etal. 2019).

Extant research on communication research patterns has also addressed issues around

its interdisciplinary foundations. For instance, So (1988) found that communication is one

of the less diverse ﬁelds amongst social sciences, and Smith (2000) also discovered very

limited diversity while examining the interdisciplinarity of technical communication jour-

nals. Speciﬁcally focusing on Journal of Communication, Park and Leydesdorﬀ (2009)

found there was little citation activity for disciplines other than communication. However,

as Zhu and Fu (2019) argue, these studies were limited in many ways:

Their research scopes were not suﬃciently broad enough to reﬂect the intellectual

diversity of the entire ﬁeld of communication, barely focusing either on shortlisted,

top-tier journals (excluding emerging and niche research areas) or on a speciﬁc

period of time (ignoring the time-evolving nature of the ﬁeld). The ﬁndings mainly

oﬀer descriptive information, but not analytical investigations into the possible asso-

ciations, which thereby conﬁnes the research implications (Zhu and Fu 2019, p. 279).

Other scholars investigated speciﬁc patterns in communication publication trends. For

instance, by analyzing the publication patterns of nine leading journals, Freelon (2013)

established the main topics, methods, and citation universes of the ﬁeld, empirically dem-

onstrating that, in communication research, better-known journals tend to publish work that

is quantitative, empirical, epistemologically social-scientiﬁc, and American in nature. The

major caveat in this spread is that it almost certainly underrepresents work that is “quali-

tative, purely theoretical, critical, and non-American” (Freelon 2013, p. 22). Thus, what

holds for methodological diversity presumably holds for epistemic and thematic diversi-

ties, too. Freelon also implemented descriptive statistics to account for such research pat-

terns, complemented with social network analyses. Freelon’s ﬁndings have been recently

extended by Günther and Domahidi (2017), who analyzed the main themes of top-tier

journals in communication and found less thematic diversity than expected. Günther and

Domahidi (2017) implemented a topic modelling to specify the myriad of topics that artic-

ulate communication research, implicitly deﬁning diversity as the distribution of frequen-

cies for each variable under analysis.

Scientometrics

1 3

Leydersdorﬀ and Probst (2009) considered communication studies as a hybrid research

ﬁeld between political science and social psychology. The authors analyzed cross-citations

between journals in all three ISI categories. They found that, with the development of

the strength and identity of communication studies as a genuine discipline, the border of

communication with social psychology has become sharper than the border with political

science.

Besides the analysis of general publication patterns and the interdisciplinary founda-

tions in the ﬁeld, there is a tradition of scholarship that deals with diversity measures in

diﬀerent segments of the global academy in general, and in communication in particular

(Hendrix etal. 2016). Walter etal. (2018) analyzed many aspects of diversity through the

examination of articles published in Journal of Communication from 1951 to 2016. The

study concentrated mostly on diversities in terms of methodology, interdisciplinary per-

spectives and theoretical foci. Diversity measures thus far were assessed by calculating per-

centages of diﬀerent research categories, statistically describing the research tendencies of

the ﬁeld.

More recently, Zhu and Fu (2019) analyzed all the SSCI indexed communication jour-

nals with respect to interdisciplinarity. Their study focuses on the longitudinal citation

records of communication journals over the past two decades (1997–2016), in order to

measure the amount of citations to and from diﬀerent research ﬁelds. Speciﬁcally, Zhu and

Fu (2019) estimate the diversity of knowledge transfer (including knowledge import and

knowledge export) regarding the ﬁeld of communication. Their method was inspired by

network science. Outward citations were measured by out-degree centrality, while inward

citations were measured by weighted in-degree centrality. In addition, Zhu and Fu’s (2019)

study also measured the longitudinal correlation between citation metrics and journal

impact factor (JIF), showing that, besides a growing absolute interdisciplinarity, commu-

nication scholarship has been faced with stagnant relative interdisciplinarity over the years.

In contrast to former studies, while most typically concentrate on a sole aspect of diver-

sity, like citation patterns (Bunz 2005), interdisciplinarity (Park and Leydesdorrﬀ 2009;

Zhu and Fu 2019), or methodological and topical foundations of the ﬁeld (Freelon 2013;

Günther and Domahidi 2017), our study explores and reports multiple variables that

account for the holistic vision of the ﬁeld’s diversity. Hence, research diversity is not cali-

brated as a discrete dimension, but as a complex system made of 15 diﬀerent variables that

extant research has examined separately (Walter etal. 2018). As opposed to former studies

that mostly calculate research diversity through descriptive statistics (i.e. frequencies and

percentages), our study provides robust mathematical equations and a systematic research

protocol aimed at both assessing diversity claims in science and inferring both the evolu-

tion and current state of diﬀerent intellectual terrains.

Gatekeeping andgeopolitics: measuring thegeographical diversity ofeditorial

boards andauthors

The diversity within the editorial boards of communication journals and its related-

ness to publication trends and patterns of the ﬁeld have been also widely studied. Extant

research has demonstrated that the discipline is far from being diverse in terms of edito-

rial boards’ geographical diversity, and most scholarship has pointed to a signiﬁcant West-

ern and especially American dominance in this body of governance (Lauf 2005; Demeter

2018; Goyanes 2020). Leeds-Hurwitz (2019) adds that the diversity of editorial boards

might correlate with the journals’ production model. The author assumes that, at least in

Scientometrics

1 3

communication, open access, especially diamond open access journals, might have a more

diverse editorial board than journals under the classic production scheme. Youk and Park’s

(2019) study examined the geographical diversity and publication patterns of editors and

editorial board members in communication journals, showing that the diversity of editorial

boards was related to the journal’s aﬃliated association (NCA and ICA), international ori-

entation, and interdisciplinary nature.

The geopolitical diversity of communication journals has also been widely investigated

in the last decade (Bunz 2005; Chakravartty etal. 2018; Demeter 2018; Goyanes and Dem-

eter 2020). Ganter and Ortega (2019) argue that, while there is an increasing diversity in

communication journals germane to certain Latin-American topics, leading Western jour-

nals and conferences are still lacking diversity in terms of Latin-American authors. The

geopolitical diversity and intraregional imbalance were measured by descriptive statistics,

through which the authors identify, proportionally, the participation of diﬀerent world

regions in the European communications community. Guenther and Joubert (2017), ana-

lyzed both gender and geopolitical diversity in science communication journals throughout

time, ﬁnding that although gender inequalities have decreased slightly, Western dominance

remained at a similar level over the years. They measured diversity by analyzing cross-cul-

tural and cross-country collaborations, providing descriptive data on the most productive

countries in the ﬁeld of science communication (i.e. frequencies).

While the aforementioned studies made meaningful contributions towards a better

understanding of the long-standing imbalances that exist both in authorship and editorial

boards in the ﬁeld of communications, extant research does not problematize nor provide a

robust yardstick to evaluate the ﬁeld’s diversity. As a result, diversity ﬁndings are reported

in a “diversity vacuum”. Additionally, since most studies rely on descriptive statistics

(Bunz 2005) or deployed Simpson’s diversity indices (Lauf 2005; Demeter 2018), they fail

in estimating a benchmark level of diversity to contrast diversity claims in communication

studies. Our study provides computable deﬁnitions of research diversity and postulates dif-

ferent potential benchmark levels to statistically infer the state and evolution of diversity in

academic ﬁelds.

Problem statement

This brief recapitulation on how diﬀerent bibliometric studies have approached diversity

in communication hints to the fact that diﬀerent diversities—in authorship, thematic focus,

methodology, interdisciplinarity and so forth—might exist. However, the methodological

approaches and research procedures deployed by extant research were mostly based on

descriptive statistics of some speciﬁc variables, precluding us to delve deeper into the mul-

tidimensionality of diversity and establish reliable statistical inferences about the situation

and evolution of diversity within and across academic ﬁelds. In short, what extant research

lacks is a sound yardstick to empirically test diversity claims and infer the potential diver-

sity gaps that exist within academia. When does a given scientiﬁc ﬁeld have statistically

signiﬁcant diversity, and how can we establish statistical inferences on its state and evolu-

tion? Moreover, how can diﬀerent scientiﬁc ﬁelds be statistically measured to yield sound

diversity comparisons? This study seeks to address these gaps by providing a mathemati-

cally constructed formula with the direct vision to gauge diversity in communication and

statistically infer its position germane to a given benchmark population.

Scientometrics

1 3

Problematizing, dening andmeasuring research diversity: aprotocol

To follow, we present a methodological protocol to measure the research diversity of a

given ﬁeld and the material published (i.e. papers). Although in this study we focus on a

representative sample of JCR journals in Communication Sciences, the protocol and the

variables measured are both robust and wide enough to transpire onto other scientiﬁc ﬁelds

and units of analysis.

The starting point is a dataset, which is a representative sample of a given population,

whose rows are the cases to be evaluated and whose columns are the variables. The proto-

col to evaluate research diversity is based on four steps:

1. Establish the benchmark: Select the hypothesized marginal probability distributions

for all variables. In absence of other information, discrete uniform distribution may be

chosen.

2. Select a proper distance function to evaluate the discrepancy between the empirical

marginal distribution and the hypothesized. In this work we have chosen Hellinger

distance, although other distances (dissimilarities, divergence measures, indexes, etc.)

between two probability distributions may be used.

3. Compute variable diversity and ﬁeld diversity as explained below.

4. Express any research question of interest as a test of hypothesis and use the proposed

statistics based on variable diversity and ﬁeld diversity to solve the test. To obtain the

probability distributions of the test statistics (conﬁdence intervals) implement a row-

wise bootstrap in order to preserve the multivariate structure of the data. This may be

of importance in case variables are not independent.

In what follows we detail the steps of the protocol. In order to calibrate the position of

a given ﬁeld in terms of research diversity, we must design a benchmark. We labeled this

benchmark diversity pattern, for which we consider two possible situations: grounded truth

and known/given diversity. First, in the absence of other information, we assume that a

grounded truth exists when a given variable has the same proportion or frequencies in each

of its values. In terms of Probability Theory, the concept of grounded truth is known as

discrete uniform probability distribution (Everitt and Skrondal 2010). For instance, when

measuring the gender representation of a given ﬁeld in terms of ﬁrst authorship, grounded

truth will exist when 50% of the production is authored by male scholars and 50% by

female scholars. Second, a known/given diversity will exist when we know the current

diversity of a given population, or when we have established it theoretically. For instance,

measuring the gender representation of a given ﬁeld in terms of ﬁrst authorship, a known/

given diversity will exist when (a) we know the frequencies for the gender distribution of a

given benchmark population (the world, USA, a continent, the International Communica-

tion Association (ICA), all communication scientists, etc.) or (b) when we establish the fre-

quencies for the gender distribution that we theoretically assume to be diverse, for instance

55–45%, 60–40% or 90–10%.

Given that the values were unknown for most of our variables, we took grounded truth as a

benchmark and, in the remainder section, we problematize its conceptualization. We assume

that grounded truth, if actually exists, is very diﬃcult to concur, since any given journal has

its priorities, agendas, expectations and research focus that drive it to employ speciﬁc research

methodologies, focusing on specialized thematic areas. In addition, according to Knobloch-

Westerwick and Glynn (2013), there are gender-oriented topics in Communication Sciences,

Scientometrics

1 3

meaning that some thematic areas are more prone to be built and thus consumed by male

or female scholars respectively. Luck might also play a crucial role during the peer-review

process, journal selection and data gathering. Geographic imbalances might also have a sig-

niﬁcant impact on diversity, since as previous studies have demonstrated, Western geographies

dominates both research production and editorial boards (Lauf 2005; Demeter 2018), which

might suggest that their expectations, agendas and perspectives are crucial to shape communi-

cation theory, research and teaching (Curran and Park 2000; Luthra 2015).

Grounded truth serves as an ideal measure, not only to account for the potential impact

of luck, but also for the combination of external and internal variables (voluntary or not).

These conditions, however, point to potential imbalances and thus the lack of diversity

that might exist in the academy. Imbalances in a given ﬁeld are the product of internal

and external forces that struggle for domination and not the result of the selected distribu-

tion. However, due to the signiﬁcant impact that external and internal forces might have

in diversity measures, the abovementioned priorities, expectations, orientations, focus, etc.

clearly reduce the odds of accounting for a grounded truth. This means that not all values

of a given variable hold the same odds in reality, although they potentially have the same

odds of being selected. Therefore, the diﬀerent social and/or organizational agents who

discretionally and/or voluntary decide which approach or orientation is worth pursuing in

a particular journal are crucial in calibrating diversity and thus mitigating or amplifying

the distance from the grounded truth (diversity gaps). This voluntary and/or discretional

orientation is beyond chance or luck, precluding us to make value judgments and open nor-

mative discussions on how a given scientiﬁc ﬁeld should or must be (the contrary would

happen with known/given diversity, since the frequencies are known or given). Our results

simply point to how distant or close a given variable or ﬁeld is from its respective ideal,

calibrating whether this distance is statistically signiﬁcant or not. Some variables and ﬁelds

will arguably be more close to their ideal, suggesting that diversity issues are more social-

ized. Based on this preliminary problematization, we propose ﬁve diﬀerent deﬁnitions for

calibrating and comparing research diversity, according to the main objective of the meas-

urement involved. This is translated into the following mathematical terms:

Let

{

X1,X2,…,X

p}

be a set of categorical or discrete variables (available from pub-

lished papers) and let

ci1,ci2…,cik

be the diﬀerent categories or values taken by variable

Xi

, for

i=1, …,p

. Consider the following deﬁnitions:

Grounded truth We say that variable

Xi

has grounded truth if

Xi

follows a discrete uni-

form probability distribution, that is

For example, if variable

Xi

is measuring ﬁrst author’s gender (with

ci1=1

for male and

ci2=2

for female), grounded truth represents the same probability for males and females to

be the ﬁrst authors of a study in the ﬁeld of communication. Or, in other, more mathemati-

cally precise words, we say that there is grounded truth in gender if Eq.1 is

In the case that the diversity pattern is known or given, Eq.1 becomes

with

∑k

i=1p0

ij

=

1

.

(1)

p

i=

P

Xi=ci1

,P

Xi=ci2

,…,P

Xi=cik

=

1

k

,

1

k

,…,

1

k

pi=

P

Xi=1

,P

Xi=2

=

1

2

,

1

2.

p

i

=

(

P

(

X

i

=c

i1)

,P

(

X

i

=c

i2)

,…,P

(

X

i

=c

ik))

=

(

p

0

i1

,p

0

i2

,…,p

0

ik)

Scientometrics

1 3

The set

{

X

1

,X

2

,…,X

p}

serves as diversity pattern if each

Xi

has grounded truth (or

follows a known/given diversity distribution), for

i=1, …,p

. In the case of known/given

diversity, note that to establish potential tests, the known/given diversity must be a case,

scenario or context, and not the population.

Diversity of a g-group of papers This equation is oriented to calibrate the diversity of

a given group of papers with regards to several variables. In particular, the equation esti-

mates how far each variable of interest is from grounded truth. The aim is to compute a

distance between the empirical frequencies (calculated from the group of papers) and the

theoretical probability (given in Eq.1), storing all the distances in a vector. Mathemati-

cally, the diversity of a group of papers is deﬁned as a vector of distances

(dg

1

,dg

2

…,dgp)

,

where

f

i

=

(

f

i

1,…,f

ik)

being the vector with the empirical relative frequency distribution of vari-

able

Xi

in the g-group of papers,

pi

the discrete uniform probability distribution (same as

before) and

d

any distance function between discrete probability distributions. Note that

one should not compute the empirical relative frequency distribution for only one paper.

Thus, the quantity deﬁned in Eq.2 should be computed for a group

ng

of papers (

ng

≥ 10).

Also note that the vector

(dg1,dg2…,dgp)

contains the

p

distances between the empirical

relative frequency distribution of variable

X′

is

in the g-group and the corresponding dis-

crete uniform probability distribution.

g-group mean diversity This is a scalar measure to summarize the diversity of a given

group of papers, taking the mean of the elements of vector

(dg1,dg2…,dgp)

. Mathemati-

cally, g-group mean diversity is deﬁned as the mean of the

p

distances

dg

1

,dg

2

…,dgp

, that

is:

Variable diversity This measure is analogous to the diversity of a g-group of papers,

but with the diﬀerence that the whole sample of papers is considered, instead of only

measuring a small group of them. Variable diversity is deﬁned as the vector of distances

(

d

G1

,d

G2

…,d

Gp)

, where

G

is the representative sample of indexed published papers in the

research ﬁeld of interest. In our application,

G=283

, which is the number of papers that

were randomly selected as a representative sample of the Communication Sciences ﬁeld.

Field diversity This measure is analogous to g-group mean diversity, but com-

puted on the whole sample of papers, computing the mean of the elements of vector

(dG

1

,dG

2…

,dGp)

. Field diversity is deﬁned as the mean of the

p

distances

dG

1

,dG

2…

,dGp

,

that is:

We illustrate the previous concepts and deﬁnitions in Figure A1 (see the Online Appen-

dix for detailed information). Following, we describe the protocol to measure the research

diversity of a given ﬁeld. First, scholars interested in applying our diversity measurements

need to select a representative and random sample of published papers in the research area

of interest, and then a set of variables

{

X1,X2,…,X

p}

to be measured on each paper. In our

application, we have selected a representative, proportional sample of 283 JCR articles in

(2)

dgi

=d

(

f

i

,p

i)

, for i=1, …,p

,

(3)

d

g=

1

p

p

∑

i=1

d

gi

(4)

d

G=

1

p

p

∑

i=1

d

Gi

Scientometrics

1 3

Communication Sciences and 15 diﬀerent variables to measure diversity (see the coding

book below). Remember that all variables should be categorical or discrete. For each vari-

able

Xi

, authors need to compute the grounded truth or known/given diversity using Eq.1.

In our case, we compute the grounded truth for all variables, except for ﬁrst author origin/

aﬃliation and ﬁrst author gender, for which we assume the true probability distributions

given by ICA.

To measure the statistical distance between two probability distributions, authors need

to select a statistical function. In our application, we have used the Hellinger distance

(Nikulin 1994), which is related to the Bhattacharyya coeﬃcient (Bhattacharyya 1943).

Given two discrete probability distributions

P

=

(

p

1

,p

2

,…,p

k)

and

Q

=

(

q

1

,q

2

,…,q

k)

,

the Hellinger distance between P and Q is given by

In the application, we have computed the Hellinger distance between the empirical rela-

tive frequency distribution of

Xi

and the corresponding discrete uniform distribution (in the

case of grounded truth), that is, for i = 3,…,p, we have computed

Since we assume a known/given diversity in the case of variables

X1

and

X2

, Eq. 5

becomes

The distance takes values in the [0, 1]-interval, being 0 when variable

Xi

has the diver-

sity pattern (grounded truth or known/given pattern). Note that distance functions like

the Euclidean or Manhattan do not make sense here, since they do not take into account

that

∑k

j=

1fij =

1

. This approach can complement other studies, where other metrics, such

as Kulback–Leibler divergence, entropy, Simpson’s diversity, Rao–Stirling index, among

other, are used. After selecting a proper statistical distance function, authors need to com-

pute the variable diversity and store it in a row vector, and then compute the ﬁeld diver-

sity using Eq. 4. Finally, a bootstrap is needed for the estimation of g-group diversity

and g-group mean diversity. First, authors need to bootstrap the representative sample of

indexed published papers in order to obtain

B

groups of

n

papers, that is, select randomly

B

groups of

n

papers. In the application, we have taken

B=200

and

n=10

. It is important to

select groups randomly in order to avoid biased estimations.

Using Eq.2, authors have to compute the g-group diversity, for

g=1, …,B

. Then, they

must store each g-group diversity as a row of a

B×p

matrix. Call the diversity-matrix to this

matrix. Note that each column of the diversity matrix contains the bootstrap distribution of

distance

dgi

(

i=1, …,p

), that is, the bootstrap distribution of the distance of variable

Xi

to the

grounded truth distribution. Therefore, any summary statistic can be computed on these distri-

butions. We recommend obtaining the corresponding means and medians in order to compare

them to the corresponding variable diversity. Finally, using Eq.3, authors have to compute

(5)

d

(P,Q)=

1−

k

i=1

piqi

d

fi,pi

=

1−

k

j=1fij

1

k

d

fi,pi

=

1−

k

j=1fijp0

ij

Scientometrics

1 3

g-group mean diversity,

dg

, for each row of the diversity-matrix and, lastly, consider the mean

over all the bootstrap samples

as an estimation of the g-group mean diversity within the ﬁeld.

Methodology

We will now describe the methodology and protocol of data gathering and data analysis in

detail, which must be followed in order to correctly implement our diversity measurements.

First, the interested scholars need to create a pool of research papers from all manuscripts that

have been published in a given year. In our case we select 2017, Communication Sciences and

the SSCI list of Web of Science (NJ= 84). Then, authors need to make a proportional random

sample of the pool of articles that is representative to all research papers published with a mar-

gin error of ± 5%. The random selection can be implemented by using a computerized random

number generator. In our case the proportional random sample was N = 283.

After the sample selection, independent coders need to content analyze the articles under

study. In our case, we follow the Cohen kappa inter-coder agreement coeﬃcient (Cohen,

1960), which adjusts for the proportion of agreements that take place. This was evaluated

using the guidelines outlined by Landis and Koch (1977), where the strength of the kappa

coeﬃcient is as follows: 0.01–0.20 slight; 0.21–0.40 fair; 0.41–0.60 moderate; 0.61–0.80 sub-

stantial; 0.81–1.00 almost perfect. The analysis provided an inter-rater reliability of 97% and a

kappa coeﬃcient of 0.93. Therefore, the inter-coder reliability was almost perfect. All discrep-

ancies between coders must be resolved through discussion.

Finally, authors need to create a coding book (see the Online Appendix for detailed infor-

mation). In order to design and apply the set of diversity measurements previously deﬁned,

one must ﬁrst establish a set of variables which can be oriented to measure the myriad of

diversities that might exist in a given ﬁeld. In our case, we review previous literature on com-

munication research patterns and bibliometric analysis. We consider this stream of research

to be crucial in shedding light on diversity issues in Communication Sciences. Although its

main purpose is not to calibrate research diversity in the ﬁeld, it has established reliable meas-

urements to evaluate the evolution of the ﬁeld, thus indirectly providing relevant variables to

shed light on diversity issues (Freelon 2013; Günther and Domahidi 2017; Walter etal. 2018;

Demeter 2018, etc.).

All the selected articles were coded manually, since SCI/JCR do not provide data on

most of the categories and variables studied. It means that the coders downloaded the ran-

domly selected articles, and manually collected data on authors

{

X

1

,X

2

,…,X

4}

and articles

{

X

5

,X

6

,…,X

15}

. As a consequence, all the selected articles were content analyzed manu-

ally, justifying why it was impossible to conduct “big data” analysis (Gil de Zuniga and Diehl

2017). That is also he main reason to implement a proportional random sample.

d

dB =

1

B

B

∑

i=1

dg

Scientometrics

1 3

Application inthecommunication sciences eld

In Table 1 we give the variable and ﬁeld diversity according to the set of variables

{

X

1

,X

2

,…,X

15}

.1 The interpretation is as follows: grounded truth is 0 (100% diversity)

and thus values closer to 0 are more diverse than those farther from 0. As we can observe

in Table1, most variables are close to 0, the ﬁrst author gender (

X2

) being the closest and

X4

(First author aﬃliation type) the farthest oﬀ. The ﬁeld diversity is 0.2212, i.e. 77.9%

diverse.

Regarding variable diversity (see Figure A2 in the Online Appendix for details), we

observe that its values are always lower than the median (and the mean) values of the cor-

responding g-group diversities. Indeed, as the number of papers per group increases, the

g-group diversity value gets closer to variable diversity (see TableA1 in the Online Appen-

dix for detailed information).

Our initial analysis indicates some descriptive statistics of research diversity in Commu-

nication Sciences in terms of the general ﬁeld and the variables under study. However, this

scrutiny does not provide any empirical evidence regarding the existence of statistically

signiﬁcant diﬀerences between grounded truth or the known/given diversity pattern and

the ﬁeld of Communication Sciences (RQ1). Similarly, it is important to evaluate possible

statistically signiﬁcant diﬀerences between the diversity of each variable under study and

grounded truth or the known/given diversity pattern (RQ2); and between the ﬁeld (RQ3)

and each variable (RQ4) diversity in 1997 and grounded truth or the known/given diversity

Table 1 Variable and ﬁeld

diversity Category Variable diversity (distance to

the diversity pattern)

% of diversity

X10.1499 85.0

X20.0234 97.7

X30.1810 81.9

X40.4864 51.4

X50.1777 82.2

X60.0740 92.6

X70.1111 88.9

X80.3898 61.0

X90.1530 84.7

X10 0.2224 77.8

X11 0.2748 72.5

X12 0.2849 71.5

X13 0.2681 73.2

X14 0.3142 68.6

X15 0.2078 79.2

Field diversity 0.2212 77.9

1

X1=

First author aﬃliation;

X2=

First author gender;

X3=

First author ethnicity;

X4=

First author aﬃli-

ation type;

X5=

Type of authorship;

X6=

Form of collaboration;

X7=

Interdisciplinarity;

X8=

Area of

data collection;

X

9

=

Methodologies;

X10 =

Research approach;

X11 =

Type of samples;

X12 =

Paradigms;

X13 =

Content area;

X14 =

Analytical focus;

X15 =

Theoretical framework.

Scientometrics

1 3

pattern. Finally, in order to ascertain how the discipline has evolved over time, the paper

also seeks to clarify whether there are statistically signiﬁcant diﬀerences between the ﬁeld

diversity in 1997 and the ﬁeld diversity in 2017 (RQ5); and between each variable diversity

in 1997 and each variable diversity in 2017 (RQ6), and how each diversity variable ranked

according to its contribution in mitigating or amplifying diversity gaps between 1997 and

2017 (RQ7).

As a result, we collect data of the same variables under study 20year ago, following

both the methodological procedures and protocols as previously outlined. Therefore, we

representatively and randomly select (at 5% margin error) the articles published (N = 263)

in all JCR journals in “communication” in 1997 (NJ= 36). Based on our research ambi-

tions, and applying the previously deﬁned equations, we aim to answer the following

research questions:

Results

RQ1 can be solved by conducting a hypothesis test, to which the null hypothesis is

H0

∶𝜇

(

d

G)

=

0

, where

𝜇(

d

G)

is the expectation (that is, the population mean) of the dis-

tance between the ﬁeld diversity and the diversity pattern (grounded truth or the known/

given pattern). Our proposal is to test the null with the following test statistic:

whose distribution under the null can be obtained by bootstrap. We derived the distribu-

tion of the test statistic from B = 20,000 bootstrap samples of size n = 283 (see Figure A3

in the Online Appendix for a kernel estimation of the density function and TableA2 for the

conﬁdence intervals for

dG

).2 As we may observe, none of them contain the value 0, which

means that we should reject the null. However, we must point out that this null hypothesis

H0

∶𝜇

(

d

G)

=

0

is a very restrictive one, since it implies that there is grounded truth or

a known/given diversity pattern in each variable. The explanation is as follows: since a

distance cannot take negative values, a sum of distances is equal to zero if, and only if, all

the summands are equal to zero. If we look more carefully at the 99%-conﬁdence interval,

we can observe that the ﬁeld diversity is between 0.2133 and 0.2401, meaning that the

distance from the diversity pattern (grounded truth or known/given diversity pattern) is

between 21.33 and 24.01%, which is not much. Indeed, this conﬁdence interval indicates

that, in 2017, the ﬁeld diversity is between 76 and 78.7%.

To answer RQ2, we can conduct p goodness-of-ﬁt tests, with null hypothesis

H0i

∶

(

f

i1

,…,f

ik)

=

(

p0

i1

,…,p0

ik)

, for i = 1, 2 and

H

0i∶

fi1,…,fik

=

1

k

,…,1

k

, for

i = 3,…,p. In short, we test if the variables under study follow a known/given probability

distribution or a grounded truth (uniform distribution). Therefore, those variables with a p

value below 0.05/15 = 0.0033 (using Bonferroni correction) are not signiﬁcant (i.e. are not

diverse), while those above 0.0033 are statistically signiﬁcant and thus diverse. Note that,

if a signiﬁcance level of 0.01 is preferred, then this threshold becomes 0.01/15 = 0.00,067.

(6)

d

G=

1

p

p

∑

i=1

d

Gi

2 Remind that all bootstrap procedures are done case-wise in order to preserve the multivariate structure of

the data, which may be of importance if variables are not independent.

Scientometrics

1 3

In this case, we have conducted the Chi square goodness-of-ﬁt test.3 Results are shown in

Table 2, where we can observe that variable diversity is statistically signiﬁcant only for

First author gender (X2). Therefore, only this variable follows the diversity pattern, while

the others do not. We also show the 99%-conﬁdence intervals obtained by bootstrap. We

observe that Form of collaboration (X6) and Interdisciplinarity (X7) are not far from

grounded truth.

RQ3 can be solved analogously to RQ1. Speciﬁcally, we are interested in testing

H0

∶𝜇

(

d1997

G)

=

0

, where

𝜇(

d1997

G)

is the expectation (that is, the population mean) of the

distance between the ﬁeld diversity in 1997 and the diversity pattern (grounded truth or

the known/given pattern). As before, we use the test statistic of Eq.6, whose distribution

under the null is obtained by bootstrap. We derived the distribution of the test statistic from

B = 20,000 bootstrap samples of size n = 263 (see Figure A4 in the Online Appendix for

a kernel estimation of the density function and TableA3 for the conﬁdence intervals for

dG

). Since none of them contain the value 0, we reject the null, meaning that in 1997 the

ﬁeld was not 100% diverse. Indeed, the 99%-conﬁdence interval indicates that the ﬁeld

diversity is between 0.2837 and 0.3197, meaning that the distance from the diversity pat-

tern (grounded truth or known/given diversity) is from 28.37 to 31.975%. Thus in 1997, the

ﬁeld diversity was between 68 and 71.6%, around 7 points lower than in 2017.

RQ4 can be solved analogously to RQ2, that is, conducting p goodness-of-ﬁt tests, one

for each variable. As before, we performed the Chi square goodness-of-ﬁt test.4 Results are

Table 2 Results of the Chi

square goodness-of-ﬁt test and

99%-conﬁdence interval

Category Chi square statistic p value 99%-CI (boot-

strap)

% of

diver-

sity

range

X185.1651 0.0000 0.1122 0.2154 78 89

X21.2497 0.2636 0.0000 0.0773 92 100

X368.2721 0.0000 0.1288 0.2362 76 87

X4526.2721 0.0000 0.4405 0.5475 45 56

X564.1449 0.0000 0.1282 0.2329 77 87

X611.7809 0.0028 0.0239 0.1306 87 98

X726.0106 0.0000 0.0604 0.1676 83 94

X8451.8587 0.0000 0.346 0.4698 53 65

X942.2933 0.0000 0.1064 0.2188 78 89

X10 105.7845 0.0000 0.172 0.2808 72 83

X11 138.3004 0.0000 0.2251 0.3319 67 77

X12 198.4629 0.0000 0.2349 0.3409 66 77

X13 182.8445 0.0000 0.2223 0.3269 67 78

X14 293.9293 0.0000 0.2685 0.3688 63 73

X15 224.9894 0.0000 0.1709 0.2607 74 83

4 Note that, all expected cell values are greater than 5, hence no Yates correction is needed. For example, if

we compute expected cell values in the worst case, which are those corresponding to variable “area of data

collection” with k = 13 categories, we have that for a sample size of n = 263, they are n·1/k = 20.23.

3 Note that all expected cell values are greater than 5, hence no Yates correction is needed. For example, if

we compute expected cell values in the worst case, which are those corresponding to variable “area of data

collection” with k = 13 categories, we have that for a sample size of n = 283, they are n·1/k = 21.77.

Scientometrics

1 3

shown in Table3, where we reject the null for all variable diversities at any signiﬁcance

level. Therefore, none of them are 100% diverse. Looking at the 99%-conﬁdence intervals,

we can see that First author gender (X2) is the closest to the diversity pattern.

To solve RQ5 we have to check if the diﬀerences between the ﬁeld diversity in 1997 and

the ﬁeld diversity in 2017 are statistically signiﬁcant. Thus, we can perform a test with null

hypothesis

H

0∶𝜇

(

d

1997

G)

=𝜇

(

d

2017

G)

. Our proposal is to test the null with the following

test statistic:

whose support is the interval [− 1, 1]. The distribution of the statistic under the null is com-

puted from B = 20,000 bootstrap samples of sizes n1 = 263 and n2 = 283 (see Figure A5 in

the Online Appendix for a kernel estimation of the density function and TableA4 for the

conﬁdence intervals). We can observe that both limits are positive, indicating that the ﬁeld

diversity in 2017 is closer to the diversity pattern (grounded truth or known/given pattern)

than in 1997. Since both limits are positive (zero is not included in the conﬁdence interval),

we conclude that there are statistically signiﬁcant diﬀerences between the ﬁeld diversity in

1997 and 2017.

To answer RQ6, we proceed analogously to RQ5 and obtain 15 bootstrap conﬁdence

intervals in order to test if there are statistically signiﬁcant diﬀerences between each vari-

able diversity in 1997 and 2017. We propose the following tests statistics:

with support in [− 1, 1]. Their distributions are obtained from B = 20,000 bootstrap samples

of sizes n1 = 263 and n2 = 283. Table 4 contains the conﬁdence intervals (see Figure A6

Diﬀ_ﬁeld

=d1997

G

−d2017

G

Diﬀ_variable

(i)=d

1997

Gi

−d

2017

Gi

,for i =1, …

, 15.

Table 3 Results of the Chi

square goodness-of-ﬁt test and

99%- conﬁdence interval

Sample 1997

Category Chi square statistic p value 99%-CI (boot-

strap)

% of

diver-

sity

range

X1113.7046 0.0000 0.1660 0.2884 71 83

X250.1736 0.0000 0.0991 0.2109 79 90

X3172.5057 0.0000 0.2778 0.3858 61 72

X4552.3916 0.0000 0.4558 0.5702 43 54

X5128.4715 0.0000 0.2508 0.3649 64 75

X697.3308 0.0000 0.1567 0.2687 73 84

X797.2395 0.0000 0.1565 0.2684 73 84

X8945.1369 0.0000 0.4668 0.5963 40 53

X973.308 0.0000 0.1488 0.2642 74 85

X10 122.3232 0.0000 0.2462 0.3807 62 75

X11 244.0494 0.0000 0.2683 0.3792 62 73

X12 107.2053 0.0000 0.1950 0.3067 69 81

X13 161.8593 0.0000 0.2643 0.3947 61 74

X14 168.1939 0.0000 0.2954 0.4045 60 70

X15 175.4867 0.0000 0.2399 0.3512 65 76

Scientometrics

1 3

in the Online Appendix for the kernel density estimations). If we look at 99%-conﬁdence

intervals, we can see that the variables which show statistically signiﬁcant diﬀerences in

their diversity between 1997 and 2017 are: First author gender (X2), First author ethnicity

(X3), Type of authorship (X5), Form of collaboration (X6), Interdisciplinarity (X7), Land of

data collection (X8) and Theoretical framework (X15). All of them have experienced a sig-

niﬁcant diversity increase within these 20years.

As we can observe in Fig.1 (RQ7) there is a notable increase in the percentage of diver-

sity between 1997 and 2017 in the vast majority of variables. Only for research paradigm

(X12) the percentage of diversity in 1997 was greater than in 2017, but this diﬀerence was

not statistically signiﬁcant. Therefore, as the ﬁeld becomes more mature, the diversity

gaps are generally mitigated, in most cases signiﬁcantly, while the diversity gap in 1997 is

higher than in 2017 in only one case.

Theoretical applications andmore empirical testing: cross‑comparisons

betweenacademic elds

The application of our diversity measurements can also be implemented to calibrate,

compare and rank academic ﬁelds. The diﬀerent variables under study can be adapted or

complemented with other values, as long as the studied category (i.e. variable) remains

the same across all disciplines. For instance, X1 (First author origin/aﬃliation), X2 (First

author gender), X3 (First author ethnicity), X4 (First author aﬃliation type), X5 (Type of

authorship), X6 (Form of collaboration), X7 (Interdisciplinary) and X8 (Land of data col-

lection) are variables whose values should not change much across the spectrums of both

natural and social sciences. However, X9 (Methodologies), X10 (Research approach), X11

(Type of samples), X12 (Paradigms), X13 (Content area), X14 (Analytical focus), X15 (Theo-

retical framework) are variables with values that should be adapted and/or complemented

to capture the nature of each ﬁeld under study. Nevertheless, in order to make sound

Table 4 Conﬁdence intervals for

Diﬀ_variable statistic

***Stands for statistically signiﬁcant

Category 99%-CI (bootstrap)

X1− 0.0185 0.1406

X20.0543 0.1967 ***

X30.0744 0.2297 ***

X4− 0.0639 0.0988

X50.0470 0.2047 ***

X60.0571 0.2109 ***

X70.0220 0.1744 ***

X80.0368 0.2139 ***

X9− 0.0356 0.1236

X10 − 0.0015 0.1584

X11 − 0.0314 0.1205

X12 − 0.1143 0.0427

X13 − 0.0298 0.1336

X14 − 0.0448 0.1111

X15 0.0073 0.1513 ***

Scientometrics

1 3

comparisons, every variable under analysis should be added in all ﬁelds, modifying or

maintaining the values for its measurement. Therefore, when comparing academic ﬁelds,

variables must remain the same across the board, while values can be adapted, modiﬁed,

changed or complemented.

The previous application of diﬀerent diversity measurements was based on a single aca-

demic ﬁeld, i.e. communication sciences, comparing two diﬀerent points in time (current

situation vs. 20years ago). However, in this section, we apply said diversity measurement

to calibrate the diversity distance between two academic ﬁelds: communication and politi-

cal science. First, from a statistical point of view, diﬀerent academic ﬁelds (i.e. academic

ﬁeld “A” and academic ﬁeld “B”) can be considered similar to an academic ﬁeld in a par-

ticular year (i.e. 1997 or 2017). Therefore, this new scenario can be solved following previ-

ous indications, in particular those from RQ5 to RQ7. Indeed, when comparing two diﬀer-

ent academic ﬁelds, we are interested in testing

H

0∶𝜇

(

d

A

G)

=𝜇

(

d

B

G)

, thus we use the test

statistic previously proposed:

Second, to test if there are statistically signiﬁcant diﬀerences between each variable

diversity in Field A and Field B, the following test statistics are proposed:

In consequence, we compare these two diﬀerent ﬁelds. Concerning paper selection for

Political Sciences, we chose the same analogous method that we used for communication,

leading to a proportional random sample of N = 329 papers (inter-rater reliability of 95%

Diﬀ_ﬁeld

=dA

G

−dB

G

Diﬀ_variable

(i)=d

A

Gi

−d

B

Gi

,for i =1, …

, 15.

0.0% 20.0%40.0% 60.0% 80.0% 100.0%

X8

X4

X14

X3

X11

X13

X5

X10

X15

X12

X6

X7

X1

X9

X2

ﬁeld

2017 1997

Fig. 1 Diversity gaps between variables in 1997 and 2017

Scientometrics

1 3

and a kappa coeﬃcient of 0.90). Regarding the diversity pattern, we compute the grounded

truth for all variables, except for ﬁrst author origin/aﬃliation and ﬁrst author gender, for

which we assume the true probability distributions given by IPSA (International Political

Science Association).

The distributions of the previous statistics under the null are computed from 20,000

bootstrap samples of sizes n1 = 329 and n2 = 283 (see Figure A7 in the Online Appendix for

a kernel estimation of the density function and TableA5 for the corresponding conﬁdence

intervals). We can observe that both limits are positive, meaning that the ﬁeld diversity for

Communication Sciences is closer to the diversity pattern (grounded truth or known/given

pattern) than for Political Sciences. Since both limits are positive (zero is not included

in the conﬁdence interval), we conclude that there are statistically signiﬁcant diﬀerences

between both academic ﬁelds.

Concerning variable diversity, Table 5 contains the corresponding conﬁdence intervals

(see Figure A8 in the Online Appendix for the kernel density estimations). If we look at

99%-conﬁdence intervals, we can see that the variables which show statistically signiﬁcant

diﬀerences in their diversity between both ﬁelds are: First author origin/aﬃliation (X1),

First author ethnicity (X3), Form of collaboration (X6), Interdisciplinarity (X7), Methodolo-

gies (X9), Paradigms (X12), Content area (X13) and Theoretical framework (X15). In particu-

lar, Communication has more diversity than Political Sciences in First author origin/aﬃli-

ation, First author ethnicity, Form of collaboration, Interdisciplinarity, Methodologies and

Theoretical Framework; whereas the contrary occurs in the Paradigms and Content area.

Finally, as we can observe in Fig.2, the diversity in Communication is greater than that

of Political Science in eight out of ﬁfteen variables, although those diﬀerences were statis-

tically signiﬁcant in only six of them.

Table 5 Conﬁdence intervals

for Diﬀ_variable statistic for

the comparison between two

academic ﬁelds

***Stands for st atistically signiﬁcant

Category 99%-CI (bootstrap)

X10.0464 0.1918 ***

X2− 0.0645 0.0523

X30.0412 0.1874 ***

X4− 0.1269 0.0265

X5− 0.0567 0.0885

X60.0587 0.2036 ***

X70.0737 0.2204 ***

X8− 0.1558 0.0157

X90.0114 0.1619 ***

X10 − 0.1449 0.0020

X11 − 0.0153 0.1340

X12 − 0.1857 − 0.0387 ***

X13 − 0.1796 − 0.0322 ***

X14 − 0.1381 0.0038

X15 0.1004 0.2369 ***

Scientometrics

1 3

Discussion andconclusion

The goal of this study was to propose and test a methodological protocol to calibrate the

research diversity in a given scientiﬁc ﬁeld. Speciﬁcally, we tested the mathematical fea-

sibility of our instrument within the ﬁelds of Communication and Political Sciences. This

study oﬀers three inter-related contributions regarding this line of inquiry at diﬀerent lev-

els of analysis: theoretical, methodological and empirical. First, we propose six theoreti-

cal deﬁnitions to empirically measure research diversity, describing their mathematical

and theoretical foundations in detail: grounded truth, known/given diversity, diversity of

a g-group of papers, g-group mean diversity, variable diversity and ﬁeld diversity. While

extant research in ecology (Simpson’s Index by Magurran 1988), economics (Hirschmann-

index by Hirschmann 2018) and information sciences (Shannon index by Shannon 1948)

have provided diﬀerent equations that may be applied to assess diversity in a myriad of

realms, our contribution extends these indices by designing ad hoc measurements to empir-

ically calibrate the potential and multiple dimensions of diversity in science. The 15 cat-

egories proposed are thus aimed to capture a detailed portrait of the ﬁeld diversity in Com-

munication, also adding a temporal frame for longitudinal examination.

Second, we present and describe a research protocol for a step-by-step evaluation of how

the diﬀerent measurements should be applied following standard procedures of data col-

lection and analysis. After proposing a research protocol and problematizing the potential

adaptation of our instrument to calibrate diversity in diﬀerent academic ﬁelds, we empiri-

cally apply it to evaluate the state of communication, comparing the diversity state in 2017

with the situation twenty years ago. Our empirical evidences demonstrated that diversity

should be calibrated as a complex phenomenon and thus diﬀerent dimensions must be con-

sidered. As a result, a given ﬁeld may hold almost grounded truth diversity in one cat-

egory, while still lacking it in other variables, as our results demonstrate. In addition, as

contrasted with former cross-sectional research (Lauf 2005; Demeter 2018), a longitudinal

0.0% 20.0% 40.0%60.0% 80.0%100.0%

X4

X8

X14

X12

X11

X13

X10

X15

X3

X5

X9

X1

X7

X6

X2

ﬁeld

Communicaon Sciences Polical Sciences

Fig. 2 Diversity gaps between variables in political sciences and communication sciences

Scientometrics

1 3

analysis adds a better understanding of the phenomena, addressing how diﬀerent features

of research diversity may evolve during the course of the years, also signaling potential

diversity gaps that may exist in a given ﬁeld.

In our analysis of the Communication Sciences ﬁeld, we show that, comparing it to

grounded truth or given/known diversity, most variables and the ﬁeld as a whole are not

statistically signiﬁcant (i.e. are not diverse), suggesting that the discipline still has room

for improvement at its macro and micro levels of inclusiveness. In this regard, only the

variable “ﬁrst author gender” is statistically signiﬁcant, demonstrating that the knowledge

production of communication research, taking the ICA as baseline, is representative of its

members. The longitudinal analysis also shows that the ﬁeld is improving its overall lev-

els of diversity throughout the years, as the research production in 2017 has a statistically

signiﬁcant increase in diversity compared to that of 1997. Our results thus suggest that

most scientiﬁc stakeholders aim to create a more open space for communication research,

in which diﬀerent diversity dimensions may harmoniously coexist. Finally, in order to

account empirically for cross-comparisons between scientiﬁc ﬁelds, our analysis applies

diversity measures to calibrate the diversity distance between two cousin ﬁelds: Communi-

cation and Political Sciences. Our ﬁndings show that Communication, compared to Politi-

cal Sciences, is a signiﬁcantly more diverse ﬁeld, especially in terms of ﬁrst author origin,

ethnicity, interdisciplinarity and the methods employed.

In summary, the main purpose of our study is to systematize a general and generaliz-

able protocol for measuring diversity within diﬀerent academic ﬁelds. Therefore, our main

ambition is to deﬁne a protocol that measures the diversity of a discipline in a multivari-

ate way, based on the information on their authors and the type and characteristics of the

research they carry out. Speciﬁcally, we measure the diversity of a discipline through the

analysis of a multivariate sample of articles published in JCR. For each of the variables of

interest, the distance to a reference standard or, in its absence, to the discrete uniform dis-

tribution (since we consider that a variable is more diverse the more balanced its probabil-

ity distribution) is calculated. Our protocol is assumed to be general enough to be applica-

ble to other disciplines.

This study has some limitations that should be addressed by future research. First, while

we aimed to be consistent with the categorization schema of former studies (Lauf 2005;

Demeter 2018), the geographical coding could be diﬀerent, nuancing the ﬁnal results. Sec-

ond, and most importantly, in order to establish our benchmark comparisons, we rely on

grounded truth when frequency distributions were unknown and on given/known diver-

sity when such data was potentially available (in our case from ICA or IPSA for gender

and geographical diversity). While our measurements work well and provide sound results

for comparisons (between years and across ﬁelds), as the benchmark is always the same

(although it is not perfect) for gauging the diversity of a given ﬁeld in a given point of

time (i.e. 2017), results may change according to the benchmark of selection. A potential

solution to establish a more reliable benchmark for given/known diversity when studying

scientiﬁc ﬁelds in a given point of time is to content analyze a more open scientiﬁc ranking

(Scopus) and then adjust the frequencies for each variable to the data gathered from JCR

journals.

Raising the level of diversity in the global academy in general, and in communication

studies in particular, has been a topic of emerging interest in the last decades. The dis-

cussions concerning the internalization and diversiﬁcation of the ﬁeld are rife with both

empirical analyses (Lauf 2005; Demeter 2018; Toth 2018) and theoretical polemics (Wais-

bord and Mellado 2014; Waisbord 2019), while an inferential examination of research

diversity in Communication Studies has been missing. This article contributes to current

Scientometrics

1 3

discussions on research diversity by providing a mathematical apparatus and research pro-

tocol for diversity calibration, accounting for the inherent complexity and multidimension-

ality of the phenomenon and its potential adaptation to other ﬁelds. The mathematical deﬁ-

nitions proposed could be of great interest for all academics and policymakers oriented to

grasp the complexity and evolution of diversity in science, and all those stakeholders who

want to establish a more inclusive and diverse global science.

References

Agresti, A., & Agresti, B. F. (1978). Statistical analysis of qualitative variation. In K. F. Schussler (Ed.),

Social methodology (Vol. 9, pp. 204–237). New York: Wiley.

Bhattacharyya, A. (1943). On a measure of divergence between two statistical populations deﬁned by their

probability distributions. Bulletin of the Calcutta Mathematical Society, 35, 99–109.

Bone, F., Hopkins, M. M., Ráfols, I., Molas-Gallart, J., Tang, P., Davey, G., & Carr, A. M. (2019). DARE to

be diﬀerent? Applying diversity indicators to the evaluation of collaborative research projects. Science

Policy Research Unit—SPRU working paper series 2019–09, University of Sussex, UK.

Borgman, C. L. (1989). Bibliometrics and scholarly communication: Editor’s introduction. Communication

Research, 16(5), 583–599.

Boschma, R. (2005). Proximity and innovation: A critical assessment. Regional Studies, 39(1), 61–74.

Bunz, U. (2005). Publish or perish: A limited author analysis of ICA and NCA journals. Journal of Commu-

nication, 55(4), 703–720. https ://doi.org/10.1111/j.1460-2466.2005.tb030 18.x.

Chakravartty, P., Kuo, R., Grubbs, V., & McIlwain, C. (2018). #CommunicationSoWhite. Journal of Com-

munication, 68(2), 254–266. https ://doi.org/10.1093/joc/jqy00 3.

Cohen, J. (1960). A coeﬃcient of agreement for nominal scales. Educational and Psychological Measure-

ment, 20(1), 37–46.

Curran, J., & Park, M. (2000). De-Westernizing media studies. London: Routledge.

Demeter, M. (2018). Changing center and stagnant periphery in communication and media studies: National

diversity of major international journals in the ﬁeld of communication from 2013 to 2017. Interna-

tional Journal of Communication, 12, 29.

Dhanani, A., & Jones, M. J. (2017). Editorial boards of accounting journals: Gender diversity and inter-

nationalisation. Accounting, Auditing & Accountability Journal, 30(5), 1008–1040. https ://doi.

org/10.1108/AAAJ-08-2014-1785.

Everitt, B. S., & Skrondal, A. (2010). The Cambridge dictionary of statistics (4th ed.). New York: Cam-

bridge University Press.

Feeley, T. H. (2008). A bibliometric analysis of communication journals from 2002 to 2005. Human Com-

munication Researh, 34(3), 505–520. https ://doi.org/10.1111/j.1468-2958.2008.00330 .x.

Freelon, D. (2013). Co-citation map of 9 comm journals, 2003–2013. Retrieved May 5, 2020, from http://

dfree lon.org/2013/09/05/co-citat ion-map-of-9-comm-journ als-2003-2013/.

Funkhouser, E. T. (1996). The evaluative use of citation analysis for communication journals. Human Com-

munication Research, 22(4), 563–574. https ://doi.org/10.1111/j.1468-2958.1996.tb003 79.x.

Ganter, S. A., & Ortega, F. (2019). The invisibility of Latin American Scholarship in European media and

communication studies: Challenges and opportunities of de-westernization and academic cosmopoli-

tanism. International Journal of Communication, 13, 68–91.

Gil de Zuniga, H., & Diehl, T. (2017). Citizenship, social media, and big data: Current and future research

in the social sciences. Social Science Computer Review, 35(1), 3–9.

Gini, C. (1912). Variabiliti e Mutabiliti. Studi Economicoaguridici della facotta di Giurisprudenza dell.

Cagliari: Universite di Cagliari III, Parte II.

Goyanes, M. (2020). Editorial boards in communication sciences journals: Plurality or standardization?

International Communication Gazette, 82(4), 342–364.

Goyanes, M., & Demeter, M. (2020). How the geographic diversity of editorial boards aﬀects what is pub-

lished in JCR-ranked communication journals. Journalism & Mass Communication Quarterly. https ://

doi.org/10.1177/10776 99020 90416 9.

Griﬃn, D. J., Bolkan, S., Holmgren, J. L., & Tutzauer, F. (2016). Central journals and authors in commu-

nication using a publication network. Scientometrics, 106(1), 91–104. https ://doi.org/10.1007/s1119

2-015-1774-4.

Scientometrics

1 3

Guenther, L., & Joubert, M. (2017). Science communication as a ﬁeld of research: Identifying trends,

challenges and gaps by analysing research papers. Journal of Science Communication, 16(2), 1–19.

https ://doi.org/10.22323 /2.16020 202.

Günther, E., & Domahidi, E. (2017). What communication scholars write about: An analysis of 80years

of research in high-impact journals. International Journal of Communication, 11, 3051–3071.

Hendrix, K. G., Mazer, J. P., & Hess, J. A. (2016). Forum: Diversity and scholarship on instructional

communication. Communication Education, 65(1), 105–127.

Hirschman, A. O. (2018). National power and the structure of foreign trade. Berkeley: University of

California Press.

Keating, D. M., Richards, A. S., Palomares, N. A., Banas, J. A., Joyce, N., & Rains, S. A. (2019). Titling

practices and their implications in communication research 1970–2010: Cutesy cues carry citation

consequences. Communication Research. https ://doi.org/10.1177/00936 50219 88702 5.

Knobloch-Westerwick, S., & Glynn, C. J. (2013). The Matilda eﬀect—Role congruity eﬀects on schol-

arly communication: A citation analysis of communication research and journal of communication

articles. Communication Research, 40(1), 3–26. https ://doi.org/10.1177/00936 50211 41833 9.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biom-

etrics, 33(1), 159–174.

Lauf, E. (2005). National diversity of major international journals in the ﬁeld of communication. Jour-

nal of Communication, 55(1), 139–151. https ://doi.org/10.1111/j.1460-2466.2005.tb026 63.x.

Leeds-Hurwitz, W. (2019). Moving (slowly) toward understanding knowledge as a global commons.

Journal of Multicultural Discourses. https ://doi.org/10.1080/17447 143.2019.16958 06.

Leydesdorﬀ, L., & Probst, C. (2009). The delineation of an interdisciplinary specialty in terms of a

journal set: The case of communication studies. Journal of the American Society for Information

Science and Technology, 60(8), 1709–1718.

Leydesdorﬀ, L., & Rafols, I. (2010). Indicators of the interdisciplinarity of journals: Diversity, central-

ity, and citations. Journal of Informetrics, 5(1), 87–100.

Leydesdorﬀ, L., Wagner, C. S., & Bornmann, L. (2019). Interdisciplinarity as diversity in citation pat-

terns among journals: Rao–Stirling diversity, relative variety, and the Gini coeﬃcient. Journal of

Informetrics, 13(1), 255–269.

Livingstone, S. (2007). Internationalizing media and communication studies: Reﬂections on the Interna-

tional Communication Association. Global Media and Communication, 3(3), 273–288.

Luthra, R. (2015). Transforming global communication research with a view to the margins. Commu-

nication Research and Practice, 1(3), 251–257. https ://doi.org/10.1080/22041 451.2015.10791 56.

Magurran, A. E. (1988). Ecological diversity and its measurement. Princeton, NJ: Princeton University

Press.

Metz, I., Harzing, A. W., & Zyphur, M. J. (2016). Of journal editors and editorial boards: who are the

trailblazers in increasing editorial board gender equality? British Journal of Management, 27(4),

712–726. https ://doi.org/10.1111/1467-8551.12133 .

Nikulin, M. S. (1994). Hellinger distance. In Encyclopedia of mathematics. Retrieved May 5, 2020, from

https ://www.encyc loped iaofm ath.org/index .php/Helli nger_dista nce.

Paisley, W. (1989). Bibliometrics, scholarly communication, and communication research. Communica-

tion Research, 16(5), 701–717. https ://doi.org/10.1177/00936 50890 16005 010.

Park, H., & Leydesdorﬀ, L. (2009). Knowledge linkage structures in communication studies using

citation analysis among communication journals. Scientometrics, 81(1), 157–175. https ://doi.

org/10.1007/s1119 2-009-2119-y.

Ràfols, I. (2014). Knowledge integration and diﬀusion: Measures and mapping of diversity and coher-

ence. In Y. Ding, R. Rousseau, & D. Wolfram (Eds.), Measuring scholarly impact (pp. 169–190).

Cham: Springer.

Rafols, I., & Meyer, M. (2010). Diversity and network coherence as indicators of interdisciplinarity:

Case studies in bionanoscience. Scientometrics, 82(2), 263–287.

Rao, C. R. (1948). The utilization of multiple measurements in problems of biological classiﬁcation.

Journal of the Royal Statistical Society, B, 13, 159–193.

Rao, C. R. (1982a). Diversity and dissimilarity coeﬃcients: A uniﬁed approach. Theoretical Population

Biology, 21(1), 24–43.

Rao, C. R. (1982b). Diversity: Its measurement, decomposition, apportionment and analysis. Sankhya:

The Indian Journal of Statistics, Series A, 44(1), 1–22.

Reeves, B., & Borgman, C. L. (1983). A bibliometric evaluation of core journals in com-

munication research. Human Communication Research, 10(1), 119–136. https ://doi.

org/10.1111/j.1468-2958.1983.tb000 07.x.

Scientometrics

1 3

Rice, R. E., Borgman, C. L., & Reeves, B. (1988). Citation networks of communication journals, 1977–

1985 cliques and positions, citations made and citations received. Human Communication Research,

15(2), 256–283. https ://doi.org/10.1111/j.1468-2958.1988.tb001 84.x.

Rogers, E. M. (1999). Anatomy of the two subdisciplines of communication study. Human Communication

Research, 25(4), 618–631. https ://doi.org/10.1111/j.1468-2958.1999.tb004 65.x.

Rousseau, R. (2019). Correspondence. On the Leydesdorﬀ–Wagner–Bornmann proposal for diversity meas-

urements. Journal of Informetrics, 13, 906–907.

Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3),

379–423. https ://doi.org/10.1002/j.1538-7305.1948.tb013 38.x.

Smith, E. O. (2000). Strength in the technical communication journals and diversity in the serials cited.

Journal of Business and Technical Communication, 14(2), 131–184. https ://doi.org/10.1177/10506

51900 01400 201.

So, C. Y. (1988). Citation patterns of core communication journals: An assessment of the develop-

mental status of communication. Human Communication Research, 15(2), 236–255. https ://doi.

org/10.1111/j.1468-2958.1988.tb001 83.x.

Sokal, R. R., & Sneath, P. H. A. (1963). Principles of numerical taxonomy. San Francisco: Freeman.

Stephan, P. E., & Levin, S. G. (1991). Inequality in scientiﬁc performance: Adjustment for attribution and

journal impact. Social Studies of Science, 21(2), 351–368. https ://doi.org/10.1177/03063 12910 21002

007.

Stirling, A. (2007). A general framework for analyzing diversity in science, technology and society. Journal

of the Royal Society, Interface, 4(15), 707–719.

Toth, J. (2018). “U.S. journals can aﬀord to remain regional, but we can not.” Author distribution-based

internationality of Eastern European communication journals. KOME—An International Journal of

Pure Communication Inquiry, 6(2), 1–15. https ://doi.org/10.17646 /KOME.2018.21.

Waisbord, S. (2016). Communication studies without frontiers? Translation and cosmopolitanism across

academic cultures. International Journal of Communication, 10(2016), 868–886.

Waisbord, S. (2019). Communication. A post-discipline. London: Polity Press.

Waisbord, S., & Mellado, C. (2014). De-westernizing communication studies: A reassessment. Communica-

tion Theory, 24(4), 361–372. https ://doi.org/10.1111/comt.12044 .

Walter, N., Cody, M. J., & Ball-Rokeach, S. J. (2018). The ebb and ﬂow of communication research: Seven

decades of publication trends and research priorities. Journal of Communication, 68(2), 424–440. https

://doi.org/10.1093/joc/jqx01 5.

Wasserman, H. (2018). Power, meaning and geopolitics: Ethics as an entry point for global communication

studies. Journal of Communication, 68(2), 441–451. https ://doi.org/10.1093/joc/jqy00 1.

Willems, W. (2014). Provincializing hegemonic histories of media and communication studies: Toward

a genealogy of epistemic resistance in Africa. Communication Theory, 24(4), 415–434. https ://doi.

org/10.1111/comt.12043 .

Youk, S., & Park, H. S. (2019). Where and what do they publish? Editors’ and editorial board members’

aﬃliated institutions and the citation counts of their endogenous publications in the ﬁeld of communi-

cation. Scientometrics, 120(3), 1237–1260. https ://doi.org/10.1007/s1119 2-019-03169 -x.

Zhang, L., Glänzel, W., & Liang, L. M. (2009). Tracing the role of individual journals in a cross-citation

network based on diﬀerent indicators. Scientometrics, 81(3), 821–838.

Zhang, L., Janssens, F., Liang, L. M., & Glänzel, W. (2010). Journal crosscitation analysis for validation and

improvement of journal-based subject classiﬁcation in bibliometric research. Scientometrics, 82(3),

687–706.

Zhang, L., Rousseau, R., & Glänzel, W. (2016). Diversity of references as an indicator for interdisciplinarity

of journals: Taking similarity between subject ﬁelds into account. Journal of the Association for Infor-

mation Science and Technology, 67(5), 1257–1265.

Zhu, Y., & Fu, K. W. (2019). The Relationship between interdisciplinarity and journal impact factor in the

ﬁeld of communication during 1997–2016. Journal of Communication, 69(3), 273–297. https ://doi.

org/10.1093/joc/jqz01 2.

Scientometrics

1 3

Aliations

ManuelGoyanes1,2 · MártonDemeter3 · AureaGrané4 ·

IreneAlbarrán‑Lozano4· HomeroGildeZúñiga2,5,6

Márton Demeter

demeter.marton@uni-nke.hu

Aurea Grané

aurea.grane@uc3m.es

Irene Albarrán-Lozano

ialbarra@est-econ.uc3m.es

Homero Gil de Zúñiga

hgz@usal.es

1 Department ofCommunication, Carlos III University, C/Madrid 133, Madrid, Spain

2 Democracy Research Unit (DRU), Political Science, University ofSalamanca, Salamanca, Spain

3 Department ofSocial Communication, National University ofPublic Service, Budapest, Hungary

4 Department ofStatistics, Universidad Carlos III de Madrid, Madrid, Spain

5 Department ofFilm Production andMedia Studies, Pennsylvania State University, StateCollege,

USA

6 Facultad de Comunicación y Letras, Universidad Diego Portales, Santiago, Chile