ArticlePDF Available

Do ads influence rankings? Evidence from the higher education sector

Authors:

Abstract and Figures

Media outlets often produce higher education rankings. One feature of these media platforms is that they are largely financed, via advertising, by the higher education institutions they also rank. This paper investigates the relationship between university advertising in the Times Higher Education magazine and their place in the ranking published in the same magazine. Using a fixed-effect identification strategy, the analysis finds that advertising is associated with an improvement of around 15 ranks in the subsequently published world university ranking. Further analysis provide mixed evidences of a media bias. One potential explanation of these results is that advertising institutions follow better reporting practices regarding data used to build up the ranking.
Content may be subject to copyright.
Do ads influence rankings? Evidence from
the higher education sector.
Forthcoming in Education Economics
Julien Jacqmin
April 15, 2021
Abstract
Media outlets often produce higher education rankings. One feature of
these media platforms is that they are largely financed, via advertising,
by the higher education institutions they also rank. This paper investi-
gates the relationship between university advertising in the Times Higher
Education magazine and their place in the ranking published in the same
magazine. Using a fixed-effect identification strategy, the analysis finds
that advertising is associated with an improvement of around 15 ranks in
the subsequently published world university ranking. Further analysis
provide mixed evidences of a media bias. One potential explanation of
these results is that advertising institutions follow better reporting prac-
tices regarding data used to build up the ranking.
Keywords: higher education rankings, two-sided markets, advertising
.
JEL codes: L15, L82, I23, M37
NEOMA Business School. 1 Rue du Marechal Juin. 76130 Mont Saint-Aignan-France.
Email: julien.jacqmin@neoma-bs.fr. I am especially thankful for the helpful feedbacks and
inputs of Ananya Sen, Quentin David, James Dearden, Geraint Johnes and two anonymous
reviewers. All remaining errors are mine.
1 Introduction
Since the turn of the Millennium, university rankings have become more and
more prominent in the higher education landscape. Institutional leaders of
universities use ranking information as key performance indicators to guide
and assess their decisions. Governments use them to see whether they are dis-
tributing public funding adequately. Academics use them in their job search.
Students (and their parents) use them to guide their decision about where to
study. Hazelkorn, Loukkola and Zhang (2014) have empirically confirmed
claims about the highly influential role of these relative performance indica-
tors. Surveying 171 institutional leaders from 39 countries, they find that 87%
of them closely monitor their position and 61% claim setting ranking targets.
Using a natural experiment from U.S. News and World Report College Rank-
ings, Luca and Smith (2013) find that, on average, a one-rank improvement
leads to a 1% increase in the number of applicants to that university.
University league tables are far from uncontroversial. They have led to
much discussion, both public and academic. One common critique is that
rankings measure mostly research-related activities, like publication output
and the presence of top-notch researchers, rather than the quality of teach-
ing activities (Dehon, McCathie and Verardi, 2009). Research often acknowl-
edges the challenging reproduction of rankings from publicly available data
(Florian, 2007). Saisana, d’Hombres and Saltelli (2011) have discussed the
difficulty in aggregating multiple indicators into a unidimensional perfor-
mance measure and point to the high volatility of the final ranking. This
paper adds to the many critiques about university rankings discussed in the
literature. Like Dearden et al. (2019), I argue that rankings can be prone
to strategic manipulation. While their theoretical explanation relates to the
profit-maximizing goals pursued by their publishers, my argumentation fo-
cuses on how media outlets are organized. Best Colleges is published by U.S.
News & World Report, the Global MBA Rankings by The Financial Times and
the University League Tables is published by The Guardian. One key feature
of these media outlets is that they function as two-sided platforms (Tirole and
Rochet, 2003). On the one hand, media outlets sell information to audiences
who value accuracy. On the other hand, they sell eyeballs to advertisers who
value the attention of their readers (Ellman and Germano, 2009). Hence, there
is a conflict of interest because the media platforms that rank universities also
earn money from these same universities, in the form of advertising revenues.
Theoretically speaking, it is a priori unclear to what extent reputational con-
1
cerns about qualitative and objective content affect this monetary incentive to
adapt the content to better suit the advertiser in a setting where it is provided
in the form of a ranking.
In the specific case of the Times Higher Education World University Rank-
ing, this paper aims to study whether investing in advertising is associated
with an improvement of the rank of higher education institutions. For this
purpose, I mix data scraped from the Times Higher Education magazine about
their ranking and universities advertising in the printed-version of the weekly
magazine. Taking advantage of the fact that universities are ranked on a
yearly basis, I control for time- and university-invariant characteristics in or-
der to attenuate the presence of omitted variable bias. In addition, I look at the
relationship between advertising and the indicator based on which the rank-
ing is computed, as well as with its sub-indicators. Furthermore, I also study
how the main result fares with respect to different subsamples and method-
ologies.
This paper contributes to the growing literature on media economics and
the conflicts of interest raised by their multi-sided mode of organization. The
literature has first looked at whether ads buy more favorable media coverage
in the form of news articles. For example, Rinallo and Basuroy (2009) find that
Italian fashion companies advertising in magazines are more likely to be men-
tioned a second time in the editorial content. More recently, Beattie, Durante,
Knight, and Sen (2020) study whether automobile manufacturers advertising
in U.S newspapers receive favorable coverage. Focusing on car safety recalls,
which, presumably, readers are strongly interesting in, they find that manufac-
turers who advertised were less likely to be referred to in articles mentioning
these issues.
Closer to the research question studied here, this work relates to the litera-
ture analyzing the potential lack of editorial independence of media outlets in
settings where they act as a third-party certifier by providing a standardized
assessment of a set of products or services. This role mostly takes the form
of expert advice, ratings or rankings. In this way, the press discloses valuable
credence characteristics to consumers (Dranove and Jin, 2010). A first work
by Reuter and Zitzewitz (2006) examined the impact of advertising on mutual
funds recommendations. In this example from the financial sector, they found
little evidence of advertising bias. Looking respectively at the beverage and
transport sectors, Reuter (2009) and Dewenter and Heiemshoff (2014) exam-
ine the independence of wine and car ratings in the specialized press. While
the former finds a negligible influence, the latter find a significant and more
2
sizable bias. One key aspect of this present work is that, in the higher educa-
tion world, rankings are the quality disclosure mechanisms receiving the most
attention (Deming and Figlio, 2016). There, the ordering of higher education
institutions is built upon a continuous performance indicator using a statis-
tical methodology that aggregates data from various sources (the institutions
being ranked, third parties and surveys). To my knowledge, this work is the
first to both highlight and empirically asses this kind of conflict of interest in
the context of higher education.
Using a fixed-effect identification strategy, I find that paying for adver-
tising pages in the THE magazine is subsequently associated with a better
ranking in the THE World University Ranking. While between 2012 and 2018,
on average 800 institutions are ranked per year, I observe that this increase
in the ranking is of about 15 ranks the year after this investment is made.
However, further analysis find mixed evidences that this association points
towards a media bias. Suggestive evidences also highlight an alternative ex-
planation. All the universities might not be on the same page regarding the
data they provide to the THE magazine, directly or indirectly via third parties
like Scopus, in order to compute the ranking. Some follow better data report-
ing practices following advices received from the magazine after advertising
in it. Interestingly, the association observed disappears from 2017 on, the year
the ranking started to be audited by an external firm. This intervention as well
as others like being more transparent about the data and the ranking method-
ology show how it is possible to dissipate the doubts regarding the conflict of
interest discussed in this paper.
I have organized this paper as follows. In Section 2, I provide some back-
ground on the Times Higher Education World University Ranking and I de-
scribe the dataset. In Section 3, I describe the empirical strategy, while in
Section 4, I present the main results. Additional robustness checks are per-
formed in Section 5. Section 6 provides a detailed discussion about how self-
regulation and governmental intervention could diminish the conflict of inter-
est at the center of this paper. I conclude in Section 7.
2 Background and data description
In order to look at whether there is a positive association between advertis-
ing and performance of the higher education institution behind this expendi-
ture, I use data from the Times Higher Education magazine as an example.
3
Times Higher Education (hereinafter THE) is a UK-based weekly magazine
that started as a supplement of The Times newspaper. The content focuses
solely on matters related to the higher education sector and, according to the
latest figures (Times Higher Education, 2017), its website attracts 32.7 million
annual visits. Its influence is worldwide, especially since first publishing the
annual THE World University Ranking in 2004. In 2018, more than 1,200 uni-
versities located across 86 countries were ranked. The main reason why data
was collected from THE magazine lies in the presence of a printed version.
This allows to retrieve historical data about the advertising done in the maga-
zine. The other two influential world-scale rankings, the QS World University
Ranking and the Academic Ranking of World Universities, better known as
the Shanghai ranking, also publish via a media platform but operate solely
online where advertising is personalized in the sense that companies leverage
data analysis to deliver individual ad messages to readers.
For dependent variables, I use data from the THE World University Rank-
ing from 2012 to 2018. The ranking is based on the ordering of an overall per-
formance indicator given each year to each higher education institutions. Both
the relative position of schools and their absolute performance as measured by
the continuous indicator will be used as main dependent variables. The per-
formance indicator is computed from the aggregation of five sub-indicators
related to teaching, research, citations, international outlook and industry in-
come, holding for a weight of respectively 30%, 30%, 30%, 7.5% and 2.5%. I
have the information concerning the teaching sub-indicator which is based on
the results of a reputation survey (done on a sample of 20,000 scholars), the
staff-to-student ratio, the doctorate-to-bachelor ratio, the doctorate awarded
to academic staff ratio and the level of institutional income. The research sub-
indicator is also based on a reputation survey conducted by THE, on the level
of research income and on the number of publications per scholar. The ci-
tation sub-indicator is built by computing the average number of citations
per scholar, using Elsevier ’s Scopus database. The international sub-indicator
considers the share of international students and staff, as well as the share of
publications from someone from the university with at least one international
co-author. Finally, industry sub-indicator is the level of income received from
the private sector weighted by the number of scholars employed. While the
five sub-indicators are freely available online, the detailed data used to com-
pute them is not. Note also that the higher education institutions“provide
and sign off their institutional data for use in the rankings” (Times Higher Ed-
ucation, 2018). In other words, the computation of the ranking requires both
4
third-party and university participation in order to obtain the data that will
be afterwards aggregated by THE.
The first main dependent variable is rankingit. As in Aghion et al. (2010),
it is the difference between the total number of universities ranked in year t
minus the rank of university iin year t. Note that I do not know the pre-
cise place of low-ranked institutions but only the bracket where it is ranked,
for example, between 500 and 550. In this case, I assume that it has the me-
dian rank over this range, which is 525. The second main dependent variable
is Indicatorit: the absolute score based on which schools are ranked. It is a
continuous measure. The five sub-indicators based on which Indicatorit is
computed will also be analyzed separately as a dependent variable. Over the
years of data , THE grew from ranking 402 to 1258 higher education institu-
tions. Hence, I will analyze an unbalanced panel. I will further address this
issue in the robustness section.
The explanatory variable will always be related to advertising, as mea-
sured in different ways using data scraped from the printed version of THE
magazine. I consider three definitions of Advertisingit . First, I consider a 12
months time window before the publication of the yearly THE World univer-
sity ranking to compute the amount of advertisement pages done. Second, a
dummy whether or not the university advertised in the 12 months before the
ranking publication is also used. Third, I lag by one year the amount of ad-
vertising done. Note that when I record the number of advertisement pages,
I add a constant equal to one, as many observations are equal to zero, and
take its log to mitigate the presence of outliers. Overall, my main conclu-
sions hold independently from how I define my explanatory variable. From
all of my more than 5,000 higher education institutions - year observations,
7.9% of them have paid THE for advertising space. The average profile of the
advertising university tends to differ from the norm. They tend to be quite
well-ranked institutions as they have a rank that is half a standard deviation
higher than the average ranked university. Younger institutions attracting a
larger share of foreign students tend to advertise more. U.K.-based and Asian
universities advertise much more than institutions from continental Europe
and the United States.
The goal of this analysis will be to see whether Advertisingit is positively
associated with Rankingit . As rankings are based on a statistical methodol-
ogy, there are two potential interpretations behind this relationship, with dif-
ferent implications. The first explanation is that Rankingit is measuring what
it intends to measure, without being expressly favorable towards advertis-
5
ing schools. Upstream, before building the ranking when THE collects data
from the schools and third party sources, advertising schools might receive
advices on how to report adequately their information to get a better recog-
nition. These advices can concern how to categorize their staff members or
their students. Another example is by exchanging guidelines to schools on
how to retain scholars affected to several affiliations in the computation of the
citation-based indicator, as Scopus only binds an author to one single affilia-
tion (Hottenrott and Lawson, 2017). The second explanation is that Rankingit
suffers a media bias. While the previous interpretation points towards an het-
erogeneous access to key advices, this interpretation points towards a fraudu-
lent practice. Manipulation can take place at different stages when translating
the collected information into sub-indicators or when rounding decimals of
sub-indicators before aggregating them, especially for less well-ranked insti-
tutions. Another way is by targeting a non-representative group of scholars
to answer the survey used to compute the teaching sub-indicator and who
would be more favorable for some institutions. The main reason why ma-
nipulations can happen is related to transparency: (1) the information used
to build the ranking is difficult to access and to verify ex-post and (2) the ag-
gregation process of information is only partially available. Hence, finding a
positive relationship between the two main variables of the analysis is a nec-
essary but not a sufficient condition to claim the presence of a media bias in
the ranking. Unfortunately, the two mechanisms are difficult to precisely dis-
entangle, again in large part due to the lack of transparency of the ranking.
The analysis will only provide clues regarding their respective presence.
Over the period considered in the analysis, the financing model of THE
has evolved. This change is likely related to the greater digitalization of the
media sector. As shown on Figure 1, up to 2011, the magazine published lit-
tle advertising from higher education institutions. At the time, the magazine
raised revenue from paid subscriptions and the sale of space dedicated to job
openings. While less than 1% of the magazine space was dedicated to the ad-
vertising of higher education institutions in 2011, in 2018, it made up more
than 15% of the printed content. In parallel, I also observe from Figure 1 that
the number of advertising universities has multiplied six-fold over a period
of seven years. Moving from a subscription-based to an advertising-based
model raises the question of the independence of the magazine by highlight-
ing a potential conflict of interest. With the latter model, the revenue sources
are concentrated among a smaller amount of sources. These changes increase
the likeliness to see a more advantageous ranking provided to advertising
6
Figure 1: Evolution of advertising in the THE magazine.
universities by strategically distorting the way the ranking is built in favor
of some institutions and at the expense of others. However, note that more
ads can also translate into advices provided to an increasing number of ad-
vertising institutions on how to get a better recognition of their work in the
ranking.
7
Table 1: Descriptive statistics
N Mean Std. Dev. Min Max
Dependent variables:
Rankingit 4,918 396.049 305.906 0 1128
Indicatorit 3,775 41.797 19.229 12.35 96
Explanatory variables:
Advertisingit: level-12 months 4,918 0.099 0.393 0 3.761
Advertisingit: dummy-12 months 4,918 0.081 0.272 0 1
Control variables:
Ageit 4,918 4.692 0.852 1.099 7.084
Foreignit 4,918 0.138 0.113 0 0.84
Enrollmentit 4,918 9.826 0.741 6.136 14.417
Sta f fit 4,918 2.792 0.529 0 6.762
Shanghaiit 4,918 211.797 241.265 0 949
Other dependent variables:
Teachingit 4,918 32.390 15.904 8.2 95.8
Researchit 4,918 29.242 19.755 2.6 99.5
Citationsit 4,918 54.530 26.864 0.7 100
Int.outlookit 4,918 50.537 22.927 7.2 99.9
Ind.incomeit 4,811 47.492 19.147 0 100
Five variables, all varying across years and institutions, are used as con-
trol variables: Ageit,Foreignit,Enrollmentit ,Sta f fit and Shanghaiit . The first
comes from the World Higher Education Database (WHED), the next three
variables come from the self-declared data provided by the university to THE
to construct its ranking and the last one is from the Academic Ranking of
World Universities (ARWU). Ageit is the age of the institutions computed in
number of years. Foreignit is the share of foreign students enrolled in the uni-
versity, Enrollmentit is the number of students enrolled and Sta f fit is the ratio
of academic staff to the number of students enrolled. Finally, Shanghaiit is the
ranking of higher education institution iin year tin the Academic Ranking
of World Universities (ARWU), better known as the Shanghai ranking. As for
Rankingit, I take first the difference between the total number of institutions
ranked and the precise rank of the university. The inclusion of this variable
allows to proxy the fact that some institutions are more proactive than others
in their teaching and research performances, as measured by this ranking. To
mitigate the presence of some outliers, I took the log of Ageit,enrollmentit and
sta f fit. However, this does not change the key conclusions of this analysis.
8
The descriptive statistics of the eight years of data included in the main
regressions are available in Table 1. A correlation table including the variance
inflation factor computed for the benchmark regression is provided in the Ap-
pendix (see Table 1A). The small to moderate correlation coefficients and VIF
suggest that multicollinearity is not problematic in this analysis.
3 Empirical strategy
The goal of this paper is to empirically asses the link between the advertis-
ing done by universities in the THE magazine on the ranking it publishes.
To test this relationship, I use a regression analysis that takes advantage that
the THE World University ranking is repeatedly published on a yearly basis.
Thanks to a panel data analysis, I am able to control implicitly for time- and
university-invariant observables. In addition, I also explicitly consider a set of
control variables. Compared to other papers also trying to explain (some of)
the determinants of university rankings, this longitudinal approach captures
a larger source of unobserved heterogeneity and limits the presence of omit-
ted variable bias. For example, Faria et al. (2018) and Pietrucha (2018) have
a cross-sectional dataset and Marconi and Ritzen (2015) only use two years of
observations while up to 8 years of observations are considered here.
The following equation is estimated:
Dep.var.it =α0+α1advertisingit +βXit +θt+γi+eit (1)
Where Dep.var.it has two different, but related, definitions. The first, rankingit
is the rank of the university iin year tand is an ordinal measure. It is defined
as the difference between the number of universities ranked in year tand the
rank of the university iin year t. The second is Indicatorit and is a contin-
uous measure also varying across both universities and years. It is based on
this last measure, computed by the aggregation of various sub-indicators, that
each universities are ranked. Advertisingit is the explanatory variable that
measures the amount of advertising done in the 12 months time preceding
the publication of the ranking in September of year t. Hence, the main focus
of this paper is to estimate the parameter α1. The null hypothesis is that it is
equal to zero. Xit is the vector of university-level and time-varying controls.
I include year-fixed effects θtand university-fixed effects γi. Finally eit repre-
sents the error term and α0is a constant. Standard errors are clustered at the
higher education institution level to control for within institution correlations.
9
Three key methodological limitations of this approach are worth discussing
further. The first is endogeneity, the second is linked to the nature of one of
the dependent variable and the third relates to the dataset.
The decision made by universities to invest in advertising space in the THE
magazine is not exogeneous. Endogeneity occurs mainly due to the presence
of unobserved heterogeneity. The identification strategy used here benefits
from the panel structure of the dataset and the inclusion of various variables.
This approach takes implicitly and explicitly into account a wide range of as-
pects that can both affect advertisingit and Dep.var.it in order to mitigate the
presence of this omitted variable bias. For example, where the institution is
located, whether it has a medical, an engineering or a business school, its in-
stitutional status or its scholastic tradition can all have an influence on the
decision to advertise and on its performance as measured in the THE ranking.
However, these aspects tend to change slowly over time and will be implic-
itly captured by the inclusion of university fixed effects γi. Next, the ranking
methodology can change from one year to the other like the number of uni-
versities considered can increase. This kind of unobserved heterogeneity that
influences all the schools similarly in a given year is implicitly considered via
the inclusion of year fixed effects θt. These points highlight the importance of
estimates, like here, that do not rely on cross-sectional variation only. How
internationalized the institution is is also likely a determinant of advertising.
Due to the positive nature of this selection into advertising, not including a
proxy of internationalization in the model will lead to overestimate the influ-
ence of advertising on the performance measure used in the THE magazine.
To mitigate this bias, I also explicitly control for a time- and place-varying
measure of internationalization with Foreignit, the share of foreign students,
in the model.
However, I cannot set aside the existence of a third variable that could in-
fluence both the explanatory and the dependent variable. An example of such
a time-changing factor is “ambition to excellence”. Universities with a greater
ambition are more willing to advertise and to see an improvement of their
ranking. While including as a control variable the ranking of the institution
in the Shanghai ranking might partially mitigate this concern, this identifica-
tion strategy cannot set aside that other time-changing factors are also at play.
Unfortunately, it is complicated to provide more conclusive evidences on the
relationship between adversing and the ranking done by the same magazine
(or the lack thereof) by the use of instrumental variables. To qualify as an
instrument, this variable should vary across both universities and years. In
10
addition, it should be an important determinant of the advertising decision of
universities although it should not be directly related to their ranking position.
Due to the high number of different jurisdictions considered in the ranking,
more than 80 in total, it is difficult to have additional comparable data at the
university level which, in addition, varies from year to year. No proper in-
strumental variable was found. Due to the absence of exogenous variation,
this study prevents from reaching definitive conclusions on causal inference.
Hence, overall, the coefficient estimates need to be considered cautiously, in-
terpreted as partial correlations rather than than causal relationships.
The dependent variable Rankingit is the most salient measure of university
performance. However, it is non-linear as it is a rank-ordered observation. In
addition to look at the determinants of the continuous I ndicatorit based on
which the ranking of universities is done, I also do two further robustness
checks. First, I also consider a rank-ordered logit model as in Cyrenne and
Grant (2009) or Hunold et al. (2020) with similar ranking data. There are,
however, two drawbacks with this approach: only the direction of the es-
timated coefficients can be discussed, and fixed effects cannot be included.
Hence, with this non-linear approach, I cannot rule out with as much preci-
sion the presence of unobserved heterogeneity between universities as I can
with this main estimation strategy. Second, I also normalize the dependent
variable Rankingit by dividing it by the number of ranked institution in year
t. As this results in a variable that takes value between 0 and 1, I estimate a
fractional outcome regression model using a logit estimation as in Papke and
Wooldridge (2008).
The final issue relates to the data generating process. First, the dataset is
unbalanced as the number of schools ranked varies from year to year. In ad-
dition to the previous approach where I normalize the dependent variable, I
adopt an alternative approach in analyzing a balanced dataset. For this pur-
pose, I compute the same estimator on the largest balanced panel subsample
of the data. While it shrinks the number of observations (from 5,714 to 2, 150),
key results are not impacted. Second, a well-known issue related with rank-
ing data is the presence of outliers (see for example Dehon et al. (2010) or
David (2013) for a discussion). For this purpose, as a final robustness check,
an outlier robust regressor is also estimated. Overall, with these additional
approaches, I observe that these limitations of the main estimation strategy
have no influence on the main results of this analysis.
11
4 Results
I first study whether there is an association between advertising in the past 12
months in THE magazine with the position and the indicator in the ranking
of the institution in Table 2. For regression (1) to (4), the (inverted) rank of
the school is used as a dependent variable and, for regression (5) to (8), the
performance indicator based on which universities are ranked is used. In re-
gression (1) and (5), I only include university- and year- fixed effects without
any control variable. In both cases, the null hypothesis can be rejected with
99% confidence interval. I observe that investing in more advertising is sig-
nificantly and positively associated with a higher ranking position and a per-
formance indicator. Analyzing regression (2) and (6), I see that the inclusion
of control variables only marginally influences the results. Hence, the extent
of omitted variable bias related to the inclusion of these 5 control variables is
rather limited. In addition, I observe in these two regressions, that institutions
with a high staff-to-student ratio tend to also be positively associated with the
two dependent variables considered. Further on, in the analysis, regression
(2) will be the benchmark regression.
Table 2: Main results (1): Advertising and THE ranking
(1) (2) (3) (4) (5) (6) (7) (8)
Dependent Variable: Rankingi t Rankingi t Rankingi t Rankingi t Indicatorit I ndicatorit Indicatorit Indicatorit
Advertisingit 16.56*** 14.95*** 0.82*** 0.79***
(level-12 months) (4.16) (3.63) (0.22) (0.21)
Advertisingit 13.97** 0.9***
(dummy-12 months) (5.85) (0.34)
Advertisingit 12.02** 0.56***
(level-12 months-lagged) (4.85) (0.2)
Agei t -101.83 -99.57 -93.29 5.8 5.71 0.59
(79.12) (79.26) (91.63) (6.02) (6.06) (5.71)
Forei gni t 173.25*** 174.58*** 179.56** 8.67* 8.71* 9.85**
(65.89) (66.5) (70.89) (4.96) (5.02) (4.89)
Enrol lmentit 2.37 2.33 1.33 -0.71 -0.7 -0.67
(7.72) (7.75) (7.85) (0.44) (0.44) (0.41)
Sta f fit 28.77*** 28.44*** 24.39** 1.93*** 1.92*** 1.62**
(10.68) (10.72) (10.8) (0.71) (0.71) (0.68)
Shan ghaiit 0.24*** 0.24*** 0.23*** 0 0.001 0
(0.02) (0.02) (0.02) (0.001) (0.001) (0.001)
Constant 394.41*** 701.8* 692.56* 705.51 41.74*** 14.72 15.15 38.93
(0.41) (385.52) (386.18) (446.61) (0.02) (29.16) (29.31) (27.73)
University FE YES YES YES YES YES YES YES YES
Year FE YES YES YES YES YES YES YES YES
N 4918 4918 4918 4527 3767 3767 3767 3570
Adj. R20.95 0.96 0.96 0.96 0.98 0.98 0.98 0.98
Heteroskedasticity-consistent SE in parentheses are clustered at HEI level.
Statistical significance: p<0.1, ∗∗ p<0.05, ∗∗∗ p<0.01
After using a measure of the intensity of advertising, in regression (3) and
12
(6), I only look at whether there was advertising in THE magazine. For this
purpose, I use a dummy equal to one as an explanatory variable if the uni-
versity did advertise up to 12 months before the ranking publication. While
the quality of the results remains unchanged, this approach eases the interpre-
tation of the coefficient. From regression (3), I observe that, all else equal, in-
vesting in advertising in THE magazine is associated to an increase of 14 ranks
from one year to another in the THE World University Ranking, knowing that
820 institutions on average are ranked per ranking. Finally, in regression (4)
and (8), I include a one-year lag between the explanatory variable and the
dependent variable. Again, the coefficients estimated are both positive and
significant. From a methodological viewpoint, a one-year lag also has the ad-
vantage of mitigating the presence of reverse causality that could bias the con-
clusions. However, this claim is conditional on unobserved variables being se-
rially uncorrelated. Hence, these results tend to show that the (partial) correla-
tions between Advertisingit with both Rankingit and Indicatorit are persistent.
A long-lasting concomitance provides some support that THE might provide
advertising universities with advices and information on how to monitor and
improve their performance, and in fine, their rankings. These insights are not
only beneficial to improve the position of the university in the short term, as
the ranking methodology only changes marginally from year to year, and this
interpretation could explain the persistence of the association I am measuring.
Overall, this first set of results points towards a positive and significant cor-
relation between advertising and the performance of institutions in the THE
ranking.
The performance indicator I ndicatorit based on which Rankingit is estab-
lished is the weighted sum of of 5 sub-indicators. While keeping the same
independent variables as in regression (2), I instead use these continuous sub-
indicators that have a value between 0 and 100 as dependent variables in Ta-
ble 3. From the regression (9) to (13), three are also positively and significantly
correlated with advertising: teaching,research and citations. 50% of both teach-
ing and research is determined by the outcome of a reputation survey that aca-
demics have often scrutinized (see among others Bowman and Bastedo (2011)
or Bookstein, Seidler, Fieder and Winckler (2010)). While highly speculative,
this suggests that the data from this survey can be a potential source of ma-
nipulation leading to a media bias in order to favor some institutions over
others. In practice, It is possible to manipulate the survey at the design stage
of the survey, for example, by surveying academics more inclined to be fa-
vorable towards a subset of institutions, or at the analysis stage, when being
13
incorporated together to build up the sub-indicator. Unfortunately, as further
information about this survey is not publicly available, it is not possible to
explicitly test further the plausibility of these claims.
Table 3: Main results (2): Advertising and THE sub-indicators
(9) (10) (11) (12) (13)
Dependent Variable: Teaching Research Citations Int. Outlook Ind. Income
Advertisingit 0.57** 0.86*** 2.1*** 0.1 0.02
(level-12 months) (0.22) (0.27) (0.5) (0.23) (0.48)
Ageit 0.33 6.8 14.44 2.63 14.69*
(4.79) (5.64) (9.38) (4.62) (7.92)
Foreignit 1.73 2.84 -2.02 41.28*** 7.41
(2.54) (3.39) (9.78) (7.38) (6.14)
Enrollmentit -0.68 -0.98 -0.96 0.23 -1.64**
(0.56) (0.8) (0.64) (0.48) (0.69)
Sta f fit 0.36 4.58 0.07 -0.81 4.76***
(0.51) (0.68) (1.07) (0.72) (1.31)
Shanghaiit 0.002*** 0.002*** 0.01*** 0.001* -0.003**
(0.001) (0.001) (0.002) (0.001) (0.001)
Constant 35.73 -7.01 -5.79 32.13 -19.06
(22.87) (27.1) (44.46) (22.41) (38.87)
University FE YES YES YES YES YES
Year FE YES YES YES YES YES
N 4918 4918 4918 4918 4810
Adj. R20.97 0.97 0.93 0.98 0.87
Heteroskedasticity-consistent SE in parentheses are clustered at HEI level.
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
From the three significant correlations, advertising is the most strongly as-
sociated with citations, despite being objectively measurable and verifiable in
addition to be computed based on data provided by a third party institution.
One plausible explanation is that by entering into a contractual relationship
with the magazine through an advertising campaign, the magazine also pro-
vides advices on reporting in order to make sure that all citations are well
accounted for. For example, THE can give advices on how to limit misrepre-
sented institutional affiliations - as in the case of multiple affiliations of schol-
ars - in order to make sure that they are accounted for by the citation database
used by THE, now Scopus, but which was formerly Thomson Reuters. Thanks
to this kind of advices, it is possible to improve in the short run citations, as
observed by the correlation measured in regression (11) of Table 3.
14
Table 4: Main results (3): Advertising and THE ranking- subsamples
Dependent Variable: (14) (15) (16) (17) (18) (19) (20) (21) (22)
Rankingi t 2016 >2016 Asia UK EU North Am. RoW Rank>400 Rank<400
Advertisingit 9.78** 5.98 36.07*** 5.85 14.41* 0.82 25.22*** 5.88** 2.96
(level-12 months) (4.63) (5.9) (8.97) (4.08) (8.5) (5.73) (8.41) (2.92) (4.73)
Agei t -175.72 62.62 -126.21 -48.07 -86.93 -215.43** 780.87*** 161.5 -86.4
(114.9) (179.37) (167.66) (225.57) (104.58) (82.84) (250.71) (154.39) (80.61)
Forei gni t -3.44 258.54 275.35* 62.57 76.33 3.3 167.43 144.49 -35.19
(64.93) (187.54) (157.87) (189.74) (90.81) (200.57) (119.54) (112.83) (61.9)
Enrol lmenti t -13.24 5.36* -31.22 165.73*** 4.32 -41.98 -83.91* -0.95 -2.15
(15.46) (3.02) (44.52) (61.76) (5.18) (38.74) (47.31) (4.96) (12.92)
Sta f fit 20.12* 17.82 20.17 2.3 51.24*** 22.51 25.05 37.57*** -10.57
(10.29) (17.14) (25.98) (38.25) (11.8) (20.35) (36.77) (9.81) (11.83)
Shan ghaiit 0.15*** 0.22*** 0.36*** 0.06* 0.18*** 0.21*** 0.29*** 0.07*** 0.13***
(0.05) (0.03) (0.03) (0.04) (0.03) (0.05) (0.04) (0.02) (0.02)
Constant 1175.9** 57.99 990.14 -935.42 637.38 1841.86** -2365.18** -255.79 578.27
(568.56) (837.37) (752.25) (1177.98) (531.03) (634.03) (1197.52) (747.1) (395.11)
University FE YES YES YES YES YES YES YES YES YES
Year FE YES YES YES YES YES YES YES YES YES
N 2684 2026 1041 517 1433 1073 851 2432 2261
Adj. R20.95 0.98 0.94 0.98 0.97 0.98 0.92 0.97 0.87
Heteroskedasticity-consistent SE in parentheses are clustered at HEI level.
Statistical significance: p<0.1, ∗∗ p<0.05, ∗∗∗ p<0.01
On Table 4, different subsamples are considered in order to study the het-
erogeneous effect of advertising. Splitting the sample in two, almost equal
parts, before and after 2016, I see that advertising only has a positive and sig-
nificant sign on the ranking in the earlier sample shown in regression (14).
There are three plausible explanations for these contrasting results. First,
since 2017, the THE ranking has been audited by an independent firm (Times
Higher Education, 2016). Although this audit does not circumvent the lack of
transparency of the ranking, as much of the data used is not publicly avail-
able, it can still limit the presence of conflicts of interest, at least if the audi-
tors care about their reputation. As discussed in Section 2, in the most recent
years, the incentives to adapt the ranking at the benefits of advertising univer-
sities are the largest. While very speculative, this first interpretation of these
results suggests that the audit has been very effective at limiting this prob-
lem. Second, the second subsample only considers two years of observations.
Therefore, there is less room for variation across time of the ranking for the
universities. Third, in the recent years, the number of new ranking developed
by THE has boomed. Some of these rankings focus on a subset of institutions
based on geographically-defined regions, on subjects taught or focusing on
emerging institutions only. Other new rankings now focus only on teaching
related matters, on reputation or on their adequacy with sustainable goals.
It is plausible that the advertising now influences these new rankings, rather
than the “original” THE World University Ranking, especially as it now being
15
audited. Unfortunately, it is not possible to assess which one of these three
explanations prevails with the available data.
From regression (16) to (20), I consider five different groups of countries:
Asia, the U.K., Europe, North America and Rest of the World. I obtain a pos-
itive and significant coefficient for universities located in Asia and the rest of
the world. In addition, the coefficient estimated is about twice larger than in
the benchmark regression (2). This is in part due to the fact that these uni-
versities tend to be less well-ranked. For example, in 2018, Asian universities
have on average a rank of 816, which is lower than those of European and
North American universities, ranking on average respectively at 482 and 362.
With a p-value of respectively 0.12 and 0.084, advertising is not significantly
different than zero according to the 5% threshold for U.K.- and Europe-based
institutions. The fact that the coefficient estimated in the subsample of North
American universities is far from being significantly different from zero may
be due to the lower influence of the magazine in that part of the world. Only
around 5% of those universities invested in advertising in THE magazine in
2018, which is half less than the one done with Asia- or U.K.-based institu-
tions.
Finally, I separate high from low ranked institutions. In regression (20), I
observe that, if I focus on a subsample of schools ranked outside the top 400
institutions, that advertising has a positive and significant sign. This is not the
case anymore if the focus is on schools ranked within the top 400 of the THE
World University Ranking as done in regression (21). Hence, advertising is
positively associated with performance in the THE World University Ranking
for less-well ranked institutions only. Two different explanations can be at
work and they have diverging implications. First, improving the ranking of a
less-well ranked institution raises less suspicion of manipulation as the precise
rank remains unknown as only the bracket where they are rank, like between
500 and 550, is observed. Second, better-ranked schools already adopt advices
to put their data in the best light in order to shine in the rankings. This is not
as much the case for less well-ranked institutions. On the one hand, this first
explanation assumes that the ranking is biased in a fraudulent way. On the
other hand, this second explanation provides support that neither the rank nor
the indicator is biased. Rankings measure what they are suppose to measure
but, rather, that institutions differ in their awareness about good reporting
practices to THE and third parties like Scopus. Again, more data is needed
to be able to discern the extent to which these two stories hold with more
precisions.
16
5 Robustness checks
In this section, I will derive various robustness checks related to the estimation
strategy and the data generating process. Here I only provide information of
the explanatory variables considered and the complete tables are available in
the appendix. Table 5 reproduces the first half of Table 2 where Rankingit is the
dependent variable by considering a rank-ordered logit model. With this ap-
proach, I am better able to take into account the peculiarities of the ordinal de-
pendent variable. However, it has two drawbacks. First, a rank-ordered logit
model makes it more complicated to discuss the scale of the coefficients ob-
tained. Second, it is computationally burdensome to include university fixed
effects, and hence the approach considers a smaller range of unobserved time-
constant heterogeneity at the university level. Looking from regression (23) to
(26), I observe again that Advertisingit is positively associated with a higher
position in the THE ranking.
Table 5: Robustness Checks (1): Ordered logit estimator
Dep. Var.: (23) (24) (25) (26)
Rankingit
Advertisingit 0.58*** 0.29***
(level-12 months) (0.07) (0.07)
Advertisingit 0.41***
(dummy-12 months) (0.11)
Advertisingit 0.27***
(level-12 months-lagged) (0.08)
Control variables NO YES YES YES
University FE NO NO NO NO
Year FE YES YES YES YES
N 5714 5084 5084 4694
Adj. R20.02 0.14 0.14 0.14
Heteroskedasticity-consistent SE in parentheses
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
In a second serie of robustness checks, Rankingit is transformed in a ratio
by dividing it by the yearly amount of ranked institutions. As a consequence,
this new dependent variable is a ratio between 0 and 1. To consider this non-
linear and bounded distribution, a fractional outcome regression model with
two-way fixed effects is estimated. As shown in Table 6, using this new ap-
proach, the positive and significant (partial) correlation between the two vari-
ables of interest remains.
17
Table 6: Robustness Checks (2): Other dependent variable: Rankingit (ratio)
Dep. Var.: (27) (28) (29) (30)
Rankingit (ratio)
Advertisingit 0.09*** 0.09***
(level-12 months) (0.03) (0.03)
Advertisingit 0.11**
(dummy-12 months) (0.04)
Advertisingit 0.07**
(level-12 months-lagged) (0.04)
Control variables NO YES YES YES
University FE YES YES YES YES
Year FE YES YES YES YES
N 5084 5084 5084 4694
Adj. R20.31 0.31 0.31 0.31
Heteroskedasticity-consistent SE in parentheses
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
In a third serie of robustness checks, I tackle the issue related with the
unbalanced panel that is analyzed, as the number of ranked institutions is not
constant over time. For this purpose, in Table 7, I focus on the largest balanced
panel of the original dataset. It is composed of 358 institutions observed 6
years in a row between 2013 and 2018. Qualitatively similar results as in Table
2 are observed. The smaller coefficient size can be explained by the fact that
this sample does not contain many less well-ranked institutions.
Table 7: Robustness checks (3): Balanced sample
Dep. Var.: (31) (32) (33) (34)
Rankingit
Advertisingit 12*** 9.01***
(level-12 months) (3.16) (2.83)
Advertisingit 12.91***
(dummy-12 months) (4.59)
Advertisingit 9.12***
(level-12 months-lagged) (3.41)
Control variables NO YES YES YES
University FE YES YES YES YES
Year FE YES YES YES YES
N 2150 2150 2150 2150
Adj. R20.98 0.98 0.98 0.98
Heteroskedasticity-consistent SE in parentheses are clustered at HEI level.
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
18
In a last series of robustness checks, a robust analysis is done to verify
whether the results presented so far are driven by the presence of outlying ob-
servations. As discussed in Verardi and Croux (2009), identifying outliers is
more challenging in the context of fixed effects models. For this purpose, I use
an approach similar to the one described in Verardi and Wagner (2012). First,
I delete observations identified as outliers, proceeding as follows. I initially
center all variables around their median (as the mean is too sensitive in the
presence of outliers). Then an S-estimator is used to identify outliers, defined
by a robust Mahalanobis distance greater than a critical value set at 95%. Fi-
nally, outlying observations are removed and the same fixed effect regressor
as in Table 2 is fitted to the remaining observations. Following this approach,
the new sample consists of much less observations. However, as seen in Table
8, Advertisingit and Rankingit remain significantly and positively correlated.
Table 8: Robustness checks (4): Outlier robust estimator
Dep. Var.: (35) (36) (37) (38)
Rankingit
Advertisingit 28.73*** 22.1***
(level-12 months) (6.43) (5.86)
Advertisingit 28.21***
(dummy-12 months) (10.32)
Advertisingit 17.37**
(level-12 months-lagged) (6.79)
Control variables NO YES YES YES
University FE YES YES YES YES
Year FE YES YES YES YES
N 2216 2216 2216 2029
Adj. R20.96 0.96 0.96 0.96
Heteroskedasticity-consistent SE in parentheses are clustered at HEI level.
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
6 Discussion
There is a conflict of interest raised by the fact that THE magazine - which
publishes the ranking - is assessing institutions that are, increasingly, an im-
portant source of revenue via their ad spending. Based on a fixed-effect iden-
tification strategy, the data analyzed suggests that advertising is associated
with a higher position in the ranking for the institutions behind this expendi-
ture. This claim has to be taken cautiously. It is conditional on assuming that
19
there is no time changing unobserved factors unaccounted for by the model
that are positively related with the decision to advertise and the performance
as assessed by the ranking as well as no reverse causality. In addition, it is
unclear whether this result points towards a media bias due to the fact that
the reputational concerns of the magazine are not enough to compensate for
the monetary incentives derived from advertising revenues. The analysis pre-
sented here above argues also that the ranking can be measuring what it is
intended to measure but that schools deciding to advertise follow advices con-
cerning the reporting of informations to the magazine that are then processed
as part of the construction of the ranking. Overall, complementary data and
methods are needed to circumvent the various methodological limitations of
the analysis done.
THE should not be singled out for being potentially untrustworthy, as this
conflict of interest is also potentially present among other ranking providers.
For example, in parallel to the QS World University Ranking that it man-
ages, the consultancy firm QS Quacquarelli Symonds allows universities to
pay them in order to receive a 1 to 5 star rating across various aspects. The
conflict of interest underlying this rating system has sparked attention in the
popular press (see a.o. Guttenplan (2012) and Redden (2013)) and shows a
wider concern than the one the ranking empirically analyzed in this paper.
Despite many limitations, rankings are still influential among students and
other university stakeholders. They are likely here to stay. I have shown that
the reputation of the magazine might not be able to mitigate the incentive
problem faced by THE to fund itself. Therefore, it is important to think of
mechanisms that could potentially attenuate this issue. Both self-regulations
and governmental regulations can lead to improvements.
Three kinds of self-regulation apply in the context of ranking organiza-
tions. A first approach requires more transparency on the production side
of rankings. More openness about both the methodology and the data col-
lected in order to construct them is one way to increase reputational concerns.
As a result, more fine-grained research about the interpretations behind the
(partial) correlation between advertising and ranking would be possible. Un-
fortunately, the recent move of ranking institutions like THE, QS Quacquarelli
Symonds (who publishes QS World University Rankings) and ShanghaiRank-
ing Consultancy (who publishes the Academic Ranking of World Universities)
towards the data consultancy business sector makes this evolution unlikely as
ranking data are now also being monetized by them but in new ways.
The second is to audit the ranking system by a recognized third-party, as
20
done for a few years by PricewaterhouseCoopers for the THE ranking. As
already discussed, one key issue with this approach is that it shifts the rep-
utational concerns from the shoulders of the ranking organization to that of
the auditing firm. As shown when looking at data before and after this audit,
this approach can limit the potential problems highlighted here. However, as
auditing firms are also active in the consultancy business (for both ranking or-
ganizations and ranked institutions), it is unclear whether this could mitigate
completely the conflict of interest at stake, especially as this kind of audit does
not have a legally binding value.
A third approach is more radical and would require splitting THE into dif-
ferent subparts in order to isolate the ranking operation from the rest. As ar-
gued by Dearden et al. (2019), changing the objective of the ranking operation
from for-profit to not-for-profit would limit incentives to manipulate informa-
tion. A more intermediate intervention would be to single out a problematic
aspect of the ranking like the peer assessment survey that could be external-
ized and produced by an independent institution. Such changes would be
beneficial to its target readers, mostly prospective students.
Regulatory interventions can also be beneficial. Their aim is to provide al-
ternatives to current commercial rankings in order to render the overall higher
education sector more transparent. Not only can they offer useful alternatives
to the decision-making process of the sector’s various stakeholders, but they
can also potentially have a disciplining effect on incumbent rankings. Alter-
natives can take the form of a ranking or of another form of quality disclosure
mechanism. As the needs of students and other higher education stakehold-
ers (e.g., public administrators or professors) tend to be heterogeneous, vari-
ous interventions might be needed, with each targeting a different group. The
College Scorecard system targeting prospective students and implemented in
2015 in the U.S. is one such example. This system provides easily compa-
rable information along three criteria presented in a chart (with the average
national score on each criteria as the benchmark). It compares average annual
tuition, graduation rate and salary after attending the school (computed based
on past student observations). Co-initiated in 2014 by the European Commis-
sion, U-multirank compares institutions using more than 50 criteria related to
teaching, internalization, (applied) research or knowledge transfer. To ease
comparison between institutions, a letter grading scale is used. Other poten-
tial initiatives could also take advantage of the huge amount of information
acquired by governmental authorities active in the sector, including that of
various quality assurance agencies or centralized student allocation systems
21
monitoring enrollment.
Overall, this work calls for more transparency in the higher education
world and a more proactive stance by governments for facilitating its dissem-
ination in a readable and easily comparable format. Changes in this direction
would not only lead to better-informed decisions by the various stakehold-
ers of the higher education sector, but would also make universities more ac-
countable towards them (Deming and Figlio, 2016).
7 Conclusion
This paper discusses the scope for conflict of interest concerning university
rankings produced by media outlets who raise advertising revenues from in-
stitutions that they also assess via their rankings. Using data from the printed
version of THE magazine, I use a fixed-effect identification strategy that is able
to partially mitigate for the presence of an omitted variable bias. Having the
methodological limitations discussed in the paper in mind, I find that adver-
tising in the magazine is associated with an improvement of about 15 ranks in
the subsequent ranking. Although, various additional evidences also provide
support for another explanation than the presence of a media bias. Adver-
tising universities follow carefully reporting advices received from THE that
help them better shine in the ranking. However, to be able to carefully assess
and disentangle these two explanations, more research and additional data
are required. Suspicion of a media bias could be in large part lifted if THE
magazine was more transparent about the data it uses and its methodology.
This conflict of interest is not limited to the sole THE ranking. Most other
world-scale as well as country and program specific rankings are also prone
to this issue. If not via advertisement, ranking institutions also receive rev-
enues from the higher education institutions they rank via the networking
congresses they organize for higher education administrators or student fairs
where higher education institutions can rent a stand. Recent trends towards a
more data-driven management of universities have also increasingly blurred
the lines between ranking organizations and ranked institutions, as the latter
have developed data consultancy services specifically targeted to the former.
This is already true for the three main world university rankings. One key
issue with this new form of relationship is that it is not observable. This is
in stark contrast with advertising which is imprinted. Even if the conclusions
of this empirical exercise are only partial, this data has the merit to allow an
22
analysis like the one provided hereabove. With this new evolution, the need
for public interventions to let more credible alternatives emerge will be more
and more important.
23
Abbott, A. and Silles, M. (2016). Determinants of International Student
Migration. The World Economy, Vol. 39 (5), p. 621-635.
Aghion, P., Dewatripont, M., Hoxby, C., Mas-Colell, A., and Sapir, A. (2010).
The governance and performance of universities: evidence from Europe and
the US. Economic policy, vol. 25(61), p. 7-59.
Avery, C., Glickman,M. E. , Hoxby, C. and Metrick, A. (2013). A revealed
preference ranking of U.S. colleges and universities. Quaterly Journal of Eco-
nomics, Vol. 128 (1), p. 425-467.
Beattie, G., Durante, R., Knight, B. and Sen, A. (2020), Advertising Spend-
ing and Media Bias: Evidence from News Coverage of Car Safety Recalls,
Management Science, forthcoming.
Beine, M. , Noel, R. and Ragot, L. (2014). Determinants of the International
mobility of students. Economics of Education Review. Vol. 41, p. 40-54.
Beine, M., Delogu, M. and Ragot, L. (2020). The role of fees in foreign
education: Evidence from Italy. Journal of Economic Geography. Vol. 20 (2), p.
571-600.
Bookstein, F. L. , Seidler, H., Fieder, M. and Winckler, G. (2010). Too much
noise in the Times Higher Education rankings, Scientometrics. 85, p. 295-299.
Bowman, N. A. and Bastedo, M. N. (2011), Anchoring effects in world uni-
versity rankings: exploring biases in reputation scores. Higher Education. Vol.
61 (4), p. 431-444.
Broecke, S. (2015). University rankings: do they matter in the UK?. Educa-
tion Economics, 23(2), 137-161.
Chevalier, A. and Jia, X. (2016). Subject-Specific League Tables and Stu-
dents Application Decisions. The Manchester School. 84(5), p. 600-620.
Cyrenne, P., and Grant, H. (2009). University decision making and pres-
tige: An empirical study. Economics of Education Review. 28(2), p. 237-248.
David, Q. (2013). Determinants of research production at top US universi-
ties. The BE Journal of Economic Analysis & Policy, 14(1), 81-109.
Dearden, J., Grewal, R. and Lilien, G. (2019). Strategic Manipulation of
University Rankings, the Prestige Effect, and Student University Choice. Jour-
nal of Marketing Research. Vol. 56 (4), p. 691-707.
Dehon, C., McCathie, A. and Verardi, V. (2010). Uncovering excellence in
academic rankings: A closer look at the Shanghai ranking. Scientometrics,
Vol. 83(2), p. 515-524.
Deming, D. J. and Figlio, D. (2016). Accountability in US Education: Ap-
plying Lessons from K-12 Experience to Higher Education. Journal of Economic
Perspectives. Vol. 30 (3), p. 33-56.
24
Dewenter, R. and Heimeshoff, U. (2014). Media bias and advertising: Ev-
idence from a German car magazine. Review of Network Economics. 65(1), p.
77-94.
Dranove, D. and G. Z. Jin (2010). Quality disclosure and certification: The-
ory and practice, Journal of Economic Literature. Vol. 48 (4), p. 935-963.
Ellman, M. and Germano, F. (2009). What do the papers sell? A model of
advertising and media bias. The Economic Journal. 119(537), p. 680-704.
Faria, J. R., Mixon, F. G., and Upadhyaya, K. P. (2019). Alumni donations
and university reputation. Education Economics. 27(2), p. 155-165.
Florian, R. (2007). Irreproductibility of the results of the Shanghai aca-
demic ranking of world universities. Scientometrics. vol. 72 (1), p. 25-32.
Guttenplan, D. D. (2012). Ratings at a price for smaller universities. The
New York Times. December 30, 2012.
Hazelkorn, E., Loukkola, T. and Zhang, T. (2014). Rankings in institutional
strategies and processes: Impact of illusion? European University Associa-
tion. EUA Publications 2014.
Hazelkorn, E. (2015). Rankings and the reshaping of higher education: The
battle for world-class excellence. Springer.
Hottenrott, H., and Lawson, C. (2017). A first look at multiple institutional
affiliations: a study of authors in Germany, Japan and the UK. Scientometrics,
111(1), 285-295.
Huang, M. H. (2012). Opening the black box of QS World University Rank-
ings. Research Evaluation, 21(1), 71-78.
Hunold, M., Kesler, R. and Laitenberger, U. (2020). Hotel Rankings of On-
line Travel Agents, Channel Pricing and Consumer Protection. Marketing Sci-
ence, Vol. 39 (1).
Luca, M. and Smith, J. (2013). Salience in quality disclosure: evidence from
the US News college rankings. Journal of Economics &Management Strategy.
22(1), p. 58-77.
Marconi, G. (2013). Rankings, accreditations, and international exchange
students. IZA Journal of European Labor Studies, Vol. 2 (1), p. 1-22.
Marconi, G., and Ritzen, J. (2015). Determinants of international university
rankings scores. Applied Economics. 47(57), p. 6211-6227.
Papke, L.E. and Wooldridge, J. M. (2008), Panel data methods for fractional
response variables with an application to test pass rates. Journal of Economet-
rics, vol. 145, p. 121-133.
Pietrucha, J. (2018). Country-specific determinants of world university
rankings. Scientometrics. 114(3), p. 1129-1139.
25
Pinar, M., Milla, J., and Stengos, T. (2019). Sensitivity of university rank-
ings: implications of stochastic dominance efficiency analysis. Education Eco-
nomics. 27(1), p. 75-92.
Redden, E. (2013). Scrutiny of QS rankings. Inside Higher Ed. may 29, 2013.
Reuter, J. and Zitzewitz, E. (2006). Do ads influence editors? Advertising
and bias in the financial media. The Quarterly Journal of Economics. 121(1),
p.197-227.
Reuter, J. (2009). Does advertising bias product reviews? An analysis of
wine ratings. Journal of Wine Economics. 4(2), p. 125-151.
Rinallo, D. and Basuroy, S. (2009). Does advertising spending influence
media coverage of the advertiser? Journal of Marketing. 73(6), p. 33-46.
Rochet, J.C. and Tirole, J. (2003). Platform competition in two-sided mar-
kets. Journal of the European Economic Association. 1(4), p. 990-1029
Saisana, M., d’Hombres, B. and Saltelli, A. (2011). Rickety numbers: Volatil-
ity of university rankings and policy implications. Research Policy. vol. 40, p.
165-177.
Times Higher Education (2016). THE World University Rankings 2016-
2017 passes independent audit.
Times Higher Education (2017). Media Pack: Audience Demographics,
products and specifications. Last accessed : October 4rth 2019.
Times Higher Education (2018). https://www.timeshighereducation.com/world-
university-rankings/methodology-world-university-rankings-2019.
Verardi, V., and Croux, C. (2009). Robust Regression in Stata. The Stata
Journal. Vol. 9, p. 439-453.
Verardi, V. and Wagner, J. (2012). Productivity premia for German man-
ufacturing firms exporting to the Euro-area and beyond: first evidence from
robust fixed effects estimations. The World Economy. 35(6), p. 694-712.
26
Appendix
Table A.1: Correlation matrix
(1) (2) (3) (4) (5) (6) (7) (8) VIF
(1) Rankingit 1 - - - - - - -
(2) Indicatorit 0.679 1 - - - - - -
(3) Advertisingit: level-12 months 0.141 0.098 1 - - - - - 1.06
(4) Ageit 0.207 0.328 -0.098 1 - - - - 1.17
(5) Foreignit 0.367 0.519 0.189 0.129 1 - - - 1.18
(6) Enrollmentit 0.025 0.070 0.011 0.217 -0.144 1 - - 1.4
(7) Sta f f it 0.003 -0.149 0.034 -0.048 -0.048 0.329 1 - 1.21
(8) Shanghaii t 0.675 0.649 0.077 0.301 0.244 0.271 -0.129 1 2.21
Table A.2: Advertising and THE ranking: Time window of 6 months
(1) (2) (3) (4) (5) (6) (7) (8)
Dependent Variable: Rankin git Rankingit R anki ngi t Rankingi t Over allit Overal lit Overallit Overalli t
Advertisingit 24*** 22.62*** 1.03*** 1.01***
(level-6 months) (4.97) (4.46) (0.26) (0.25)
Advertisingit 28.7*** 1.45***
(dummy-6 months) (6.61) (0.36)
Advertisingit 14.26*** 0.67***
(level-6 months-lagged) (5.29) (0.24)
Control variables NO YES YES YES NO YES YES YES
University FE YES YES YES YES YES YES YES YES
Year FE YES YES YES YES YES YES YES YES
N 4918 4918 4918 4527 3767 3767 3767 3570
Adj. R20.95 0.96 0.96 0.96 0.98 0.98 0.98 0.98
Heteroskedasticity-consistent SE in parentheses are clustered at HEI level.
Statistical significance: p<0.1, ∗∗ p<0.05, ∗∗∗ p<0.01
27
Table A.3: Robustness Checks (1): Other estimation methods: Ordered logit
Dep. Var.: (9) (10) (11) (12)
Rankingit
Advertisingit 0.58*** 0.29***
(level-12 months) (0.07) (0.07)
Advertisingit 0.41***
(dummy-12 months) (0.11)
Advertisingit 0.27***
(level-12 months-lagged) (0.08)
Ageit 0.19*** 0.2*** 0.2***
(0.05) (0.05) (0.05)
Foreignit 6.1*** 6.08*** 6.4***
(0.47) (0.47) (0.49)
Enrollmentit -0.92*** -0.92*** -0.9***
(0.09) (0.09) (0.09)
Sta f fit 0.69*** 0.69*** 0.72***
(0.1) (0.1) (0.1)
Shanghaiit 0.01*** 0.01*** 0.01***
(0.00) (0.00) (0.00)
University FE NO NO NO NO
Year FE YES YES YES YES
N 5714 5084 5084 4694
Adj. R20.02 0.14 0.14 0.14
Heteroskedasticity-consistent SE in parentheses
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
28
Table A.4: Robustness Checks (2): Different dependent variable: Rankingit
(ratio)
Dep. Var.: (13) (14) (15) (16)
Rankingit (ratio)
Advertisingit 0.09*** 0.09***
(level-12 months) (0.03) (0.03)
Advertisingit 0.11**
(dummy-12 months) (0.04)
Advertisingit 0.07**
(level-12 months-lagged) (0.04)
Ageit 1.54* 1.54* 2.05**
(0.8) (0.8) (0.86)
Foreignit 0.04 0.05 0.1
(0.66) (0.66) (0.72)
Enrollmentit -0.14** -0.14** -0.15**
(0.07) (0.07) (0.07)
Sta f fit 0.05 0.05 0.07
(0.08) (0.08) (0.08)
Shanghaiit 0 0 0**
(ratio) (0.00) (0.00) (0.00)
Constant -2.86*** -8.58** -8.61** -10.91***
(0.04) (3.65) (3.66) (3.99)
University FE YES YES YES YES
Year FE YES YES YES YES
N 5084 5084 5084 4694
Adj. R20.31 0.31 0.31 0.31
Heteroskedasticity-consistent SE in parentheses
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
29
Table A.5: Robustness checks (3): Balanced sample
Dep. Var.: (17) (18) (19) (20)
Rankingit
Advertisingit 12*** 9.01***
(level-12 months) (3.16) (2.83)
Advertisingit 12.91***
(dummy-12 months) (4.59)
Advertisingit 9.12***
(level-12 months-lagged) (3.41)
Ageit -166.92 -164.11 -171.4
(110.88) (111.09) (111.64)
Foreignit 33.01 34.04 33.38
(74.37) (74.65) (74.62)
Enrollmentit -2.75 -2.86 -2.51
(19.14) (19.08) (19.06)
Sta f fit 34.84*** 34.2*** 34.76***
(12.36) (12.33) (12.41)
Shanghaiit 0.25*** 0.25*** 0.25***
(0.04) (0.04) (0.04)
Constant 547.23*** 1210.9** 1199.2** 1230.69**
(0.44) (569.8) (570.76) (573.65)
University FE YES YES YES YES
Year FE YES YES YES YES
N 2150 2150 2150 2150
Adj. R20.98 0.98 0.98 0.98
Heteroskedasticity-consistent SE in parentheses are clustered at HEI level.
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
30
Table A.6: Robustness checks (4): Outlier robust estimator
Dep. Var.: (21) (22) (23) (24)
Rankingit
Advertisingit 28.73*** 22.1***
(level-12 months) (6.43) (5.86)
Advertisingit 28.21***
(dummy-12 months) (10.32)
Advertisingit 17.37**
(level-12 months-lagged) (6.79)
Ageit -369.54** -359.6** -447.53**
(162.06) (162.27) (200.26)
Enrollmentit 40.5 38.82 34.45
(67.45) (67.8) (68.04)
Sta f fit 51.19 50.38 51.22
(44.59) (44.58) (45.59)
Shanghaiit 0.24** 0.24*** 0.23***
(0.03) (0.03) (0.03)
Constant 352*** 1522.4 1493.91 1963.3*
(0.48) (997.4) (994.37) (1147.88)
University FE YES YES YES YES
Year FE YES YES YES YES
N 2216 2216 2216 2029
Adj. R20.96 0.96 0.96 0.96
Heteroskedasticity-consistent SE in parentheses are clustered at HEI level.
Statistical significance: p<0.1, p<0.05, ∗∗∗ p<0.01
31
... Introduction Jacqmin, 2021;Lim, 2018;Ringel et al., 2020;Shahjahan et al., 2020a). While their rankings are free to access, rankers garner revenue mostly from advertising, selling ranking data and analytics, consulting, and access to events or workshops (Chirikov, 2021;Jacqmin, 2021;Shahjahan et al., 2020a). ...
... . While their rankings are free to access, rankers garner revenue mostly from advertising, selling ranking data and analytics, consulting, and access to events or workshops (Chirikov, 2021;Jacqmin, 2021;Shahjahan et al., 2020a). While the above literature notes the importance of social media, rankers' activities in engaging and/or promoting their services in social media remain empirically unexamined. ...
Article
Full-text available
Despite the ubiquity of global university rankings coverage in media and academia, a concerted attempt to investigate the role of social media in ranking entrepreneurship remains absent. By drawing on an affect lens, we critically examine the social media activities of two commercial rankers: Times Higher Education (THE) and Quacquarelli Symonds Ltd (QS). Based on an analysis of THE’s Twitter feed and QS’ Facebook page between January and June 2020, we illuminate how rankers use social media for affective storytelling to frame and sell their expertise within global HE. First, we demonstrate how THE uses Twitter to engage an audience of institutions, governments, and administrators, reinforcing universities’ increasingly aggressive behavior as market competitors. Next, we show how QS engages a student-oriented audience on Facebook, furthering the role of students as consumers. Before and during the COVID pandemic, we observed that both rankers amplified and mobilized precarity associated with performance and participation, selling hope to targeted audiences to market their expertise as solutions—a strategy that remained amidst the global pandemic. Based on our observation of the front stage of rankers’ social media activities, we argue that rankers’ deployment of social media as a form of affective infrastructure is conducive to further sustaining, diffusing, and normalizing rankings in HE globally.
... Datele empirice sugerează că răspunsul este afirmativ. Astfel, universitățile care cumpără reclamă tipărită în THE saltă, în medie, 15 locuri în clasament (Jacqmin, 2021). Pe de altă parte, un studiu efectuat pe mai mult universități din Rusia, pe perioada 2016-2021 a arătat că acele universități care în ultimii 5 ani au contractat serviciile QS de cel puțin 5 ori (cheltuind aproape 3 milioane de euro în acest sens) au săltat cu 140 de poziții în clasamentul QS comparativ cu universitățile care nu au cumpărat consultanță QS sau au făcut-o rar (Chirikov, 2021). ...
Preprint
Full-text available
În România, distrubția fondurilor de cercetare pentru universități se face cu ajutorul unui metaranking național. Cum se întocmește acest metaranking și care sunt limitările sale? Putem avea încredere în rezultatele pe care le oferă? Acest articol analizează în detaliu atât clasamentele internaționale pe care se bazează metarankingul național cât și metdologia adoptată de acesta. Rezultatele acestor analize relevă o serie de probleme fundamentale.
... They do not include empirical evidence to demonstrate the impact of conflicts of interest on ranking outcomes. One notable exception is a recent study by Jacqmin (2021) which has shown that advertising in the Times Higher Education magazine is associated with an improvement in THE World University Rankings. However, the study design does not allow identifying whether this improvement is driven by the commercial relations between the ranker and universities or by other factors. ...
Article
Full-text available
Global university rankings influence students’ choices and higher education policies throughout the world. When rankers not only evaluate universities but also provide them with consulting, analytics, or advertising services, rankers are vulnerable to conflicts of interest that may potentially distort their rankings. The paper assesses the impact of contracting with rankers on university ranking outcomes using a difference-in-difference research design. The study matches data on the positions of 28 Russian universities in QS World University Rankings between 2016 and 2021 with information on contracts these universities had for services from QS—the company that produces these rankings. The study compares the fluctuations in QS rankings with data obtained from the Times Higher Education rankings and data recorded by national statistics. The results suggest that the universities with frequent QS-related contracts had an increase of 0.75 standard deviations (~ 140 positions) in QS World University Rankings and an increase of 0.9 standard deviations in reported QS faculty-student ratio scores over 5 years, regardless of changes in the institutional characteristics. The observed distortions could be explained by university rankers’ self-serving bias that benefits both rankers and prestige-seeking universities and reinforces the persistence of rankings in higher education.
... As a university-based research centre, CWTS Leiden is clearly part of the academic world, but given that the centre is also attached to a private consultancy company with limited liability, CWTS Leiden BV, it may also be influenced by a business logic. Attesting to this phenomenon, scholars have recently drawn attention to the conflict of interest in the organizations that both rank universities and sell consulting or advertising services to them (Jacqmin 2021;Chirikov 2022). Academics are neither exempt from wearing the proverbial multiple hats. ...
Article
Full-text available
The dramatic salience of university rankings is usually attributed to a number of macro-level trends, such as neoliberal ideology, the spread of audit culture, and globalization in the broadest sense. We propose that the institutionalization of university rankings cannot be fully accounted for without a better understanding of the meso-level processes that enable it. To explore these, we zoom in on an organization called IREG Observatory (whereby IREG stands for 'International Ranking Expert Group'). Since it first emerged, in 2002, IREG has acted as a carrier of a kind of rationalized 'faith in rankings'-a faith it has laboured to justify, diffuse, and solidify through boundary work at the intersection of technocratic, managerial, academic, and commercial spheres. Drawing on the insights gained from this particular case, the article argues that the institutionalization of university rankings is not solely a matter of universities being impelled by them, but also a matter of how actors in and around the university sector collectively partake in the legitimation of the practice of ranking universities. At a more general level, our analysis potentially provides a blueprint for understanding boundary work as a meso-level process that plays an important role in the institutionalization of rankings, and other devices of evaluation.
... To a considerable extent, the measurement of internationalisation has been left to the commercial providers of the major international league tables. This poses ethical problems -the consultancy activities of the major ranking organisations may lead to conflicts of interest which in turn distort the global university rankings (Chirikov, 2021;Jacqmin, 2021). The league tables also have methodological limitations, as indicated below. ...
Conference Paper
Full-text available
The research seeks to examine the importance of economic and non-economic incentives for managers operating in non-profit sports organisations. It also focuses on how managers in such organisations view the gap between compensation and specific aspects of managerial work, such as the amount of work done, the authority and responsibility vested in them, the required competencies and the level of their personal initiative. A questionnaire survey organised among managers of Czech nonprofit sports organisations has been chosen as the research method. In total, the research group included 270 respondents. The analysis provides overall results of the survey along with more detailed results broken down by management level and parttime/full-time employees. The results show a strong preference for non-economic incentives, especially meaningful work, helping others, self-actualisation and responsibility, mission, conviction. By contrast, the respondents attributed the least importance to contributions towards insurance premiums, holiday bonuses and other services provided by the organisation at reduced prices, or for free, and holiday. The self-assessment of the interplay between compensation and various aspects of managerial work showed the respondents viewed their work as undervalued, most notably with respect to the level of responsibility, the work done and their own personal initiative. The perceived undervaluation is viewed especially strongly for volunteers in managerial positions, who, due to the nature of volunteering, collect only minimal, mostly symbolic wages. This trend especially manifests itself strongly at higher tiers of management and with part-time workers.
... To a considerable extent, the measurement of internationalisation has been left to the commercial providers of the major international league tables. This poses ethical problems -the consultancy activities of the major ranking organisations may lead to conflicts of interest which in turn distort the global university rankings (Chirikov, 2021;Jacqmin, 2021). The league tables also have methodological limitations, as indicated below. ...
Article
Ratings and rankings are omnipresent and influential in contemporary society. Individuals and organizations strategically respond to incentives set by rating systems. We use academic publishing as a case study to examine organizational variation in responses to influential metrics. The Journal Impact Factor (JIF) is a prominent metric linked to the value of academic journals, as well as career prospects of researchers. Since scholars, institutions, and publishers alike all have strong interests in affiliating with high JIF journals, strategic behaviors to ‘game’ the JIF metric are prevalent. Strategic self-citation is a common tactic employed to inflate JIF values. Based on empirical analyses of academic journals indexed in the Web of Science, we examine institutional characteristics conducive to strategic self-citation for JIF inflation. Journals disseminated by for-profit publishers, with lower JIFs, published in academically peripheral countries and with more recent founding dates were more likely to exhibit JIF-inflating self-citation patterns. Findings reveal the importance of status and institutional logics in influencing metrics gaming behaviors, as well as how metrics can affect work outcomes in different types of institutions. While quantitative rating systems affect many who are being evaluated, certain types of people and organizations are more prone to being influenced by rating systems than others.
Article
Full-text available
To create their rankings, university-ranking agencies usually combine multiple performance measures into a composite index. However, both rankings and index scores are sensitive to the weights assigned to performance measures. This paper uses a stochastic dominance efficiency methodology to obtain two extreme, case-weighting vectors using the Academic Ranking of Worldwide Universities (ARWU) and Times Higher Education (THE) data, both of which lead to the highest and lowest index outcomes for the majority of universities. We find that both composite scores and rankings are very sensitive to weight variations, especially for middle- and low-ranked universities.
Article
Full-text available
This paper examines country-specific factors that affect the three most influential world university rankings (the Academic Ranking of World Universities, the QS World University Ranking, and the Times Higher Education World University Ranking). We run a cross sectional regression that covers 42–71 countries (depending on the ranking and data availability). We show that the position of universities from a country in the ranking is determined by the following country-specific variables: economic potential of the country, research and development expenditure, long-term political stability (freedom from war, occupation, coups and major changes in the political system), and institutional variables, including government effectiveness.
Article
This article studies the determinants of international students’ mobility at the university level, focusing specifically on the role of tuition fees. We derive a gravity model from a Random Utility Maximization model of location choice for international students in the presence of capacity constraints of the hosting institutions. The last layer of the model is estimated using new data on student migration flows at the university level for Italy. We control for the potential endogeneity of tuition fees through a classical IV approach based on the status of the university. We obtain evidence for a robust and negative effect of fees on international student mobility, with an elasticity around −0.8. The estimations also confirm the positive impact of the quality of the education and support an important role of additional destination-specific variables such as host capacity, the expected return of education, the cost of living and the existence of education programs taught in English.
Article
Do mass media bias content in favor of advertisers? If so, what market conditions limit or exacerbate this bias? We examine the relationship between advertising by auto manufacturers in U.S. newspapers and news coverage of car safety recalls between 2000 and 2014. This context allows us to separate the influence of advertisers, who prefer less coverage, from that of readers, who prefer more information about the safety risks associated with the recalls. Consistent with theoretical predictions, we find that newspapers provide less coverage of recalls issued by manufacturers that advertised more regularly on their pages over the previous two years. The effect is especially pronounced for more severe recalls, which are more likely to hurt manufacturers’ reputations. Competition for readers from other newspapers mitigates proadvertiser bias, and competition for advertising by online platforms exacerbates it. We also present suggestive evidence that less news coverage of recalls is associated with more fatal car accidents. This paper was accepted by Joshua Gans, business strategy.
Article
We investigate whether online travel agents (OTAs) assign hotels worse positions in their search results if these set lower hotel prices elsewhere.
Article
We investigate whether online travel agents (OTAs) assign hotels worse positions in their search results if these set lower hotel prices elsewhere. We investigate whether online travel agents (OTAs) assign hotels worse positions in their search results if these set lower hotel prices at other OTAs or on their own websites. We formally characterize how an OTA can use such a strategy to reduce price differentiation across distribution channels. Our empirical analysis shows that the position of a hotel in the search results of OTAs is better when the prices charged by the hotel on other channels are higher. This is consistent with the hypothesis that OTAs alter their search results to discipline hotels for aggressive prices on competing channels, thereby reducing the search quality for consumers.
Article
A multiperiod, theoretical model characterizes the relationship between a publication that ranks universities and prospective students who might use this ranking to decide which university to attend. The published ranking offers information about the universities’ objective quality but also affects their prestige, which may increase student utility. This prestige effect gives the commercial publication incentive to act contrary to the best interest of the students. If a ranking created with the commonly used attribute-and-aggregate methodology creates prestige, then to maximize profit the publication needs to (1) choose attribute score weights that do not match student preferences and (2) alter those attribute score weights over time, even in the absence of changes to student preferences and/or education technology. Without a prestige effect, the publication should choose attribute score weights that match student preferences. This model also defines a student-optimal ranking methodology that maximizes the sum of the students’ utilities. The results offer insights for prospective students who use existing rankings to choose a university, as well as which ranking designs would better align with students’ preferences.
Article
In regression analysis, the presence of outliers in the dataset can strongly distort the classical least-squares estimator and lead to unreliable results. To deal with this, several robust-to-outliers methods have been proposed in the statistical literature. In Stata, some of these methods are available through the rreg and qreg commands. Unfortunately, these methods resist only some specific types of outliers and turn out to be ineffective under alternative scenarios. In this article, we present more effective robust estimators that we implemented in Stata. We also present a graphical tool that recognizes the type of detected outliers.
Article
The present study presents a formal model of the dynamics of a university’s reputation that points to the existence of a snowball effect where alumni donations raise a university’s reputation, which in turn generates additional alumni donations. Given that econometric results presented in this study confirm the model’s main findings, supporting a university’s financial development arm at optimal levels should receive thorough consideration by the university’s administration. Our model and empirical results also suggest that university administration should better assess the reputation-enhancing facets of an institution deemed integral by peer institutions, as these determine the outcome of its fundraising efforts.