ArticlePDF Available

Using massive online choice experiments to measure changes in well-being


Abstract and Figures

Significance Gross domestic product (GDP) measures production and is not meant to measure well-being. While many people nonetheless use GDP as a proxy for well-being, consumer surplus is a better measure of consumer well-being. This is increasingly true in the digital economy where many digital goods have zero price and as a result the welfare gains from these goods are not reflected in GDP or productivity statistics. We propose a way of directly measuring consumer well-being using massive online choice experiments. We find that digital goods generate a large amount of consumer welfare that is currently not captured in GDP. For example, the median Facebook user needed a compensation of around $48 to give it up for a month.
Content may be subject to copyright.
Using massive online choice experiments to measure
changes in well-being
Erik Brynjolfsson
, Avinash Collis
, and Felix Eggers
Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA 02142;
National Bureau of Economic Research, Cambridge, MA
02138; and
Faculty of Economics and Business, University of Groningen, 9747 AE Groningen, The Netherlands
Edited by Charles Bean, London School of Economics, London, and accepted by Editorial Board Member Paul R. Milgrom February 24, 2019 (received for
review September 10, 2018)
Gross domestic product (GDP) and derived metrics such as pro-
ductivity have been central to our understanding of economic
progress and well-being. In principle, changes in consumer surplus
provide a superior, and more direct, measure of changes in well-
being, especially for digital goods. In practice, these alternatives
have been difficult to quantify. We explore the potential of
massive online choice experiments to measure consumer surplus.
We illustrate this technique via several empirical examples which
quantify the valuations of popular digital goods and categories.
Our examples include incentive-compatible discrete-choice exper-
iments where online and laboratory participants receive monetary
compensation if and only if they forgo goods for predefined
periods. For example, the median user needed a compensation of
about $48 to forgo Facebook for 1 mo. Our overall analyses reveal
that digital goods have created large gains in well-being that are
not reflected in conventional measures of GDP and productivity.
By periodically querying a large, representative sample of goods
and services, including those which are not priced in existing
markets, changes in consumer surplus and other new measures of
well-being derived from these online choice experiments have the
potential for providing cost-effective supplements to the existing
national income and product accounts.
consumer surplus
digital goods
free goods
choice experiments
Digital technologies have transformed the types of goods and
services consumed in modern economies. However, our
national measurement framework for economic growth and well-
being has not fundamentally changed since its invention in the
1930s. Gross domestic product (GDP) and derivative metrics like
productivity (typically calculated as GDP/hours worked) domi-
nate discussions of economic growth and performance. In prin-
ciple, a more comprehensive approach is now feasible. By using
massive online choice experiments we can estimate changes in
consumer surplus, the primary component of economic welfare,
and thereby supplement the traditional metrics based on GDP.
GDP measures the real value of the purchases of all final
goods by households, businesses, and government. It is the most
widely used measure of economic activity and heavily influences
policymakers in setting economic objectives and enacting inter-
ventions. Some economists, policymakers, and journalists rou-
tinely use GDP as if it were a measure of well-being (1, 2).
Nonetheless, while it is a good measure of production, GDP is a
significantly flawed measure of well-being (24). [Indeed, Simon
Kuznets, who was instrumental in developing our system of na-
tional accounts, said in 1934 that the welfare of a nation can
scarcely be inferred from [GDP].In Brynjolfsson et al.*, we
more formally discuss the relationship of GDP to welfare in the
context of choice experiments.] Attempts have been made to
design alternative measures, typically focusing on measuring
subjective well-being and life satisfaction. Despite progress,
these measures remain very imprecise (5) and a survey of leading
macroeconomists indicates that we are a long way off from
reaching consensus on how to measure well-being so that they
are reliable for policymaking (6).
The traditional national accounts are especially problematic as
metrics of well-being when prices are zero and thus are absent or
largely absent from GDP (7). This is increasingly the case for
goods in the emerging digital economy because each users copy
of a digital good, such as Wikipedia and most smartphone ap-
plications, has nearly zero marginal cost and often a zero market
price. For instance, although information goods have unques-
tionably become increasingly ubiquitous and important in our
daily lives, the official share of the information sector as a fraction
of the total nominal GDP (4 to 5%) was the same in 2016 as it
was 35 y earlier. Moreover, inmany sectors (e.g., music, media, and
encyclopedias) people substitute zero-price online services (e.g.,
Spotify, YouTube, and Wikipedia) for goods with a positive price
(e.g., CDs, DVDs, and Encyclopedia Britannica). As a result, the
total revenue contributions of these sectors to GDP figures can fall
even while consumers get access to better quality and more variety
of digital goods (see SI Appendix for an exploration of when GDP
and welfare are positively correlated and when they are uncorre-
lated or even negatively correlated).
To assess changes in living standards, and by extension the
effects of policies that might affect living standards, it is neces-
sary to properly measure the welfare gains from all goods. This
includes goods without positive market prices, including many
digital goods, public goods, and environmental goods. Because
goods with zero price have zero contribution to GDP, the welfare
Gross domestic product (GDP) measures production and is not
meant to measure well-being. While many people nonetheless
use GDP as a proxy for well-being, consumer surplus is a better
measure of consumer well-being. This is increasingly true in the
digital economy where many digital goods have zero price and
as a result the welfare gains from these goods are not reflected
in GDP or productivity statistics. We propose a way of directly
measuring consumer well-being using massive online choice
experiments. We find that digital goods generate a large
amount of consumer welfare that is currently not captured in
GDP. For example, the median Facebook user needed a com-
pensation of around $48 to give it up for a month.
Author contributions: E.B., A.C., and F.E. designed research, performed research, analyzed
data, and wrote the paper.
The authors declare no conflict of interest.
This article is a PNAS Direct Submission. C.B. is a guest editor invited by the
Editorial Board.
This open access article is distributed under Creative Commons Attribution-NonCommercial-
NoDeriv atives Lice nse 4.0 (CC BY-N C-ND).
E.B., A.C., and F.E. contributed equally to this work.
To whom correspondence should be addressed. Email:
This article contains supporting information online at
*Brynjolfsson E, Collis A, Diewert WE, Eggers F, Fox KJ (2019) GDP-B: Accounting for the
value of new and free goods in the digital economy. NBER Working Paper (Nat ional
Bureau of Economic Research, Cambridge, MA). PNAS Latest Articles
gains from such goods are not properly captured in GDP sta-
tistics. Our approach uses massive online choice experiments to
measure these welfare gains. In this paper, we focus on mea-
suring welfare gains from digital goods in particular because of
the rapid pace of innovation and adoption of these goods, which
suggests that they may have a particular important effect on the
changes in living standards. For instance, the average American
spends 22.5 h per week online as of 2018 (8). Facebook,
launched in 2004, had 2.27 billion active users worldwide as of
September 2018 and the average user spent 50 min per day on
Facebook and Instagram, up from zero in 2005 (9). WhatsApp,
launched in 2009, had 1.5 billion active users worldwide as of
January 2018 (10). These digital innovations either created
completely new goods that did not exist before or replaced and
significantly improved previously existing nondigital goods. For
example, Google (11) and Wikipedia (12) have more quantity
and better-quality results than libraries and physical encyclope-
dias. Therefore, the changes in welfare gains are likely to be
larger for digital goods than for other goods which have not
changed as radically. [Brynjolfsson and Oh (13), using an alter-
native approach to estimate consumer surplus based on time
spent, find that the annual welfare gain in consumer surplus from
free internet services was significantly higher than the welfare
gain from television use.]
Consumer surplus is defined as the difference between the
consumerswillingness to pay for a good and the amount that
they actually pay. For instance, if a person were willing to pay up
to $100 for a pair of shoes but only had to pay $70, then that
person would gain $30 of consumer surplus from that trans-
action. Economists consider the changes in consumer surplus as
a measure of changes in consumer well-being (or welfare). Total
well-being includes both consumer well-being and producer well-
being. On average, producers are estimated to capture only 2.2%
of the total welfare gains from innovation, with consumers cap-
turing the remaining surplus (14). Thus, changes in consumer
surplus are a good proxy for changes in overall well-being, es-
pecially when the ratio of consumer surplus to producer surplus
is not changing rapidly. That said, for goods with market prices
and appropriate quality adjustments of these prices over time,
changes in real GDP can also be a good proxy for changes in
well-being (7). However, GDPsusefulnessasaproxybreaksdown
for goods which have a zero market price (15). [Some goods such
as WhatsApp and Wikipedia do not have any advertising revenues,
either. Other goods such as Google Search and Facebook have
advertising revenues but the welfare gains from these goods need
not be correlated with advertising revenues (16).]
Historically, changes in consumer surplus have not been
widely used as a measure of economic progress. This reflects the
fact that it has been difficult to measure consumer surplus at
scale. Measuring consumer surplus typically requires estimating
demand curves based on exogenous variations that shift the
supply curve but not the demand curve, and it has not been
practical to identify these variations using traditional market
data for large sets of goods. However, with advances in digital
technologies, it is now feasible to collect data about thousands of
goods much more easily. In this research, we stick more closely
to a traditional microeconomic framework than the subjective
well-being research and propose a way of measuring changes in
consumer surplus using experimental variation via online choice
experiments. This approach is not only applicable to free goods
and services in the digital economy but also more broadly to
conventional goods (see SI Appendix for an example of mea-
suring a nondigital good). Choice experiments provide more
flexibility than market data because they do not require nonzero
prices or market transactions to exist and are frequently applied
in contingent valuation studies. These experiments allow us to
estimate the demand curves for any good using data from
thousands of consumers that are representative of the national
population. Our approach is easily scalable and can be used to
develop a system that tracks changes in consumer surplus of
numerous goods and services in (near) real time.
Methods and Results
We implement three distinct types of choice experiments, single-
binary discrete-choice (SBDC) experiments (17), BeckerDeGroot
Marschak lotteries (BDM) (18), and bestworst scaling (BWS)
(19), and find they have similar implications (see the end of this
section and SI Appendix for details). For ease of exposition, we
will focus on the SBDC approach, which involves consumers
making a single choice among two options: whether to keep
access to a certain good or to forego the good in return for
receiving a specific amount of money. We only ask one ques-
tion per consumer and vary the price points systematically
across thousands of consumers for each experiment. We thereby
obtain willingness to accept (WTA) valuations (i.e., the monetary
compensation needed to compensate losing access to various
We illustrate the method using Facebook to measure the
consumer surplus with SBDC choice experiments. To avoid any
bias that may affect consumer choices when the options are
purely hypothetical choices, we applied the SBDC experiment in
a nonhypothetical, incentive-compatible procedure to measure
the consumer surplus. Incentive-compatible choice experiments
make responses consequential. Specifically, it is in the best in-
terest of respondents to reveal their true preferences. We asked
consumers if they would prefer to (i) keep access to Facebook or
(ii) give up Facebook for 1 mo in return for a payment of $E(in
SI Appendix we address sensitivity of the valuation depending on
the time frame and show that our approach can detect con-
sumerssensitivity toward different time frames). We varied $E
systematically across several price points ranging between $1 and
$1,000. To make the SBDC question consequential for the
consumer, we informed them that we would randomly pick 1 out
of every 200 respondents and fulfill that persons selection (i.e.,
they get the $Ecash at the end of the month after we verify that
they have not been on Facebook for the month; see SI Appendix
for more details on the experiment). We recruited a represen-
tative sample of US Facebook users from a professional market
research firm in summer 2016 and again in 2017 to measure
annual changes in consumer surplus obtained from Facebook.
Fig. 1 plots the estimated WTA demand curves, separated for
2016 and 2017. In 2016, the samples median WTA was $48.49
for giving up 1 mo of Facebook, and this valuation dropped to
$37.76 in 2017. We used bootstrapping to calculate 95% CIs for
1 5 10 50 500
% keep Facebook
E in $
0 102030405060708090100
Fig. 1. WTA demand curves for Facebook in 2016 and 2017.
| Brynjolfsson et al.
the median WTA values, that is, CI
=[$32.04, $72.24],
=[$27.19, $51.97]. Although the median WTA values
suggest a drop in value, the CIs are fairly broad so that we cannot
establish, based on the data using this sample size from 2 y,
whether the decrease in value follows a systematic trend (in SI
Appendix we address the effect on precision by using larger
sample sizes in the sensitivity analyses).
We added usage and demographic variables to further under-
stand heterogeneity in consumer surplus. The usage of Facebook
per week is a significant predictor for the value of Facebook,
providing a reality check for our approach. Consumers value
Facebook more if they spend more time on it or have more friends.
Moreover, the more they post status updates or share pictures and
videos and the more they like and comment and play games, the
more they value Facebook. Consumers who reported using Insta-
gram or YouTube value Facebook significantly less than those who
do not use these services. Therefore, Instagram and YouTube can
be considered to be substitutes for Facebook to an extent. In terms
of sociodemographics, we find significant effects for gender and
age of the respondent, as well as household income. Specifically,
for any offer of payment of $E, female respondents are more likely
to keep Facebook than male users. The same holds for older
consumers. The effects for household income are not monotonic.
Compared with low-income households, households with an in-
come between $100,000 and $150,000 perceive significantly less
value in Facebook, while households with income above $150,000
value Facebook more. Overall, these results indicate that Face-
book provides substantial value to consumers. They would require
a median compensation of $40 to $50 for leaving this service for
a month.
We extend the experiment to additional popular digital goods
in a laboratory setting in Europe. Although the sample consists
of students and is not necessarily representative of the general
population, we used a laboratory to have more control moni-
toring the usage of the goods to further explore the effects of
incentive compatibility. We find that WhatsApp, Facebook,
and digital maps on phones are highly valued by our subjects
with median compensations for losing 1 mo of access of V536,
V97, and V59, respectively. Other applications such as Insta-
gram (V6.79), Snapchat (V2.17), and LinkedIn (V1.52) are
valued an order of magnitude lower and Skype (V0.18) and
Twitter (V0.00) have very low median valuations. (Average valu-
ations or valuations for any given consumer will typically differ
from median valuations.) In follow-up interviews, respondents
reported that the strikingly high values for WhatsApp reflected
its tight integration into their daily lives for coordination with
family, friends, colleagues, schoolmates, and others and the
high compensation needed for being digitally separated from
this network.
We also ran larger-scale choice experiments on a representa-
tive sample of the US internet population using Google Surveys,
which are well-suited to implement our one-question SBDC
experiments. Although these experiments are not incentive-
compatible, we are able to access a much larger population at
a fraction of the cost and get a more precise surplus estimate.
Moreover, by focusing on relative changes between 2016 and
2017 rather than the absolute magnitude of the valuations, any
hypothetical bias from lack of incentive compatibility is likely to
be mitigated as long as the bias in one year is similar to the bias
in the following year. Thus, a change in measured valuations
should reflect a real difference. We identified the most widely
used applications and websites on various devices and combined
them into the following eight product categories: email, search
engines, maps, e-commerce, video, music, social media, and in-
stant messaging. We ran SBDC surveys for each of these cate-
gories also in summer 2016 and 2017. In these studies, we asked
consumers to consider giving up access to these categories for
1 y. The counterfactual that consumers are provided in these
studies is to lose access to all options within a category (e.g.,
all search engines or all social media) for a year. We applied
6 to 15 price levels for each product category and gathered
500 responses for each price level for each year (n=64,940).
According to the median estimates for 2017 (Table 1), search
engines ($17,530) is the most valued category of digital goods,
followed by email ($8,414) and digital maps ($3,648). One
possible reason that these values are high relative to the other
goods in our analysis is that for many people these services are
essential to their jobs, making them reluctant to give up these
goods even in exchange for high monetary values. Whats
more, we asked for the value of the entire category such that
there are no effective between-category substitutes. Because
most consumers do not directly pay for these services, almost
all of the WTA for these goods contributes toward consumer
Video streaming services (e.g., YouTube and Netflix) are
valued by consumers with a median WTA of $1,173 per year.
Some consumers do pay for some of these services. However,
these amounts are of the order of $10 to $20 per month, or $120
to $240 per year. Our measure suggests that the surplus the
median consumers receive from these goods is a 5 to 10 multiple
of what they actually pay. Recall that the payment is visible in
national accounts, but not the consumer surplus. The remaining
categories for which we estimated the median WTA are (in
descending order) e-commerce ($842), social media ($322),
music ($168), and instant messaging ($155; see SI Appendix for
CIs and additional WTA percentiles).
As a benchmark to these hypothetical SBDC experiments, we
conducted additional choice experiments based on the BWS
approach (19). BWS asks consumers to repeatedly select the best
and worst options from experimentally varied sets of alternatives.
Consumers are required to make a trade-off when deciding
which goods they perceive as most and least valuable. This may
Table 1. Median WTA estimates for most popular digital goods categories
WTA per year
2016, $
WTA per year
2017, $
95% CI WTA
per year 2016, $
95% CI WTA
per year 2017, $
nLower Upper Lower Upper
All search engines 14,760 17,530 11,211 19,332 13,947 22,080 8,074
All email 6,139 8,414 4,844 7,898 6,886 10,218 9,102
All maps 2,693 3,648 1,897 3,930 2,687 5,051 7,515
All video 991 1,173 813 1,203 940 1,490 11,092
All e-commerce 634 842 540 751 700 1,020 11,051
All social media 205 322 156 272 240 432 6,023
All messaging 135 155 98 186 114 210 6,076
All music 140 168 112 173 129 217 6,007
Brynjolfsson et al. PNAS Latest Articles
mitigate or even eliminate any systematic hypothetical bias, at
least with respect to the ordinal ranking of the choices. We used
19 digital goods, 6 nondigital goods, and 9 price points ranging
from $1 to $20,000, which consumers compared. Because we
examined the value of foregoing access to specific services or
amenities for 1 y, the price options were also expressed as
losses (foregoing a specific amount of salary for 1 y, e.g.,
earning $10,000 less for 1 y). Fig. 2 plots estimated dis-
utilities obtained by losing access to each of these goods or
earning a specific amount less for 1 y ranked from most valuable
to least valuable. The inferred price sensitivity from these results is
closer to willingness to pay (WTP), which is typically lower than
WTA. For free digital goods, the gap between WTP and WTA
can be very large. This is because consumers are not used to
paying for these goods, and, in protest, could respond with low
valuations when asked about WTP (20). However, estimating
a demand function and interpolating WTP shows very strong
correlation among BWS and SBDC valuations (correlation =
0.911), thereby providing validity to the results in Table 1.
Likewise, valuations obtained using incentive-compatible BDM
lotteries (18) for Facebook were not statistically different from
incentive-compatible SBDC experiments, as described in SI
Discussion and Conclusion
With advances in information technologies, we can now gather
data at a large scale and close to real time. In particular, massive
online choice experiments provide estimates of the value created
by specific goods, like Facebook, and have the potential to re-
invent and supplement the measurement of economic well-being
more generally. Our approach uses the increasingly ubiquitous
digital infrastructure of the internet to provide a scalable method
of measuring changes in consumer surplus induced by techno-
logical advancements through choice experiments. Through a
series of choice experiments, we find that free digital goods
provide substantial value to consumers even if they do not con-
tribute substantially to GDP. Moreover, the consumer surplus
generated by all digital goods, estimated using quality-adjusted
prices of devices (phones and computers) and their data usage
intensity (21), is numerically similar to the sum of consumer
surplus estimates generated by most popular digital goods (sum
of valuations in Table 1), thereby providing further validity of
our results.
Our approach evaluates goods for a specific time period from
the consumers perspective. This is not necessarily the same as
the goodssocial value or their values for different time periods.
For instance, some goods might have negative externalities that
hurt other peoples well-being [e.g., fake-news sharing via social
-4.5 -4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0
No access to WhatsApp for 1 year (reference)
No access to Uber for 1 year
No access to LinkedIn for 1 year
No access to Snapchat for 1 year
No access to Skype for 1 year
No access to Twier for 1 year
No access to all ride sharing services for 1 year
No access to Instagram for 1 year
No access to Wikipedia for 1 year
Earning $1 less for 1 year
No access to public transportaon for 1 year
Earning $5 less for 1 year
No access to airline travel for 1 year
No breakfast cereal for 1 year
Earning $10 less for 1 year
No access to music streaming for 1 year
No access to Facebook for 1 year
No access to video streaming for 1 year
No access to online maps for 1 year
Earning $100 less for 1 year
No access to online shopping for 1 year
Earning $500 less for 1 year
No access to a smartphone for 1 year
No access to all email services for 1 year
No access to all search engines for 1 year
No TVs in my home for 1 year
Not meeng friends in person for 1 year
Earning $1,000 less for 1 year
No access to personal computers for 1 year
No access to all Internet for 1 year
Earning $5,000 less for 1 year
Earning $10,000 less for 1 year
Earning $20,000 less for 1 year
No toilets in my home for 1 year
Fig. 2. (Dis)Utility according to BWS.
| Brynjolfsson et al.
media (22)], negative effects on the usersown mental or physical
health, which users do not fully appreciate (23), or varying de-
grees of lock-in. Of course, the same can be true for any goods
purchased in markets, as reflected in GDP and related statistics.
An advantage of the choice-experiment approach is that it is
possible to make externalities or other aspects of user choices
salient, vary time periods, and observe the resulting effect on the
valuations. Thus, we can better understand the distinctions be-
tween private and social valuations or short- and long-term val-
uations. For example, we find that the median valuation for
giving up Facebook for 2 wk is 2.7 times the median valuation for
giving it up for 1 wk, and the median valuation for giving up
Facebook for 1 mo is 4.5 times the median valuation for giving it
up for 1 wk, suggesting a nonlinear relationship between valua-
tions and time periods. This positive nonlinear relationship is
confirmed when keeping the cash amount fixed and varying only
the time period (see SI Appendix for more details on the effect of
time periods on valuations).
Our method is highly scalable and relatively inexpensive.
Market research products such as Google Surveys let us reach
representative samples of internet populations, and a single re-
sponse to an SBDC experiment costs as little as 10 cents.
Therefore, we can run these SBDC experiments at frequent,
regular intervals to track changes in consumer surplus for many
(digital) goods and categories. This measure provides valuations
at a more detailed level than aggregate measures of subjective
well-being and can be an important complementary indicator of
consumer well-being for the digital economy. Moreover, we can
also use the same approach to estimate the welfare gains for
physical goods (e.g., in SI Appendix we give the example of
breakfast cereal) and other nonmarket goods (such as environ-
mental goods and public goods provided by the government).
Because GDP is a measure of production and not welfare, this
can help address an important and long-standing gap in our
understanding of the economy.
Hypothetical choice experiments, as shown above using
Google Surveys, can be easily and cheaply conducted online, but
the stated preferences might suffer from hypothetical bias.
However, the differences between hypothetical and incentive-
compatible approaches are much less severe when analyzing
annual changes in valuations, rather than absolute levels. GDP
growth is often considered to be more relevant to policymakers
than absolute levels. Similarly, changes in consumer surplus
valuations across time are more relevant than absolute valua-
tions. To address concerns of bias associated with answering
hypothetical questions, we also conducted incentive-compatible
choice experiments for Facebook using a representative sample
of the internet population and other popular digital goods in a
laboratory setting. Incentive-compatible choice experiments are
harder to conduct online but they provide accurate estimates of
revealed preferences.
A major limitation of our study remains the relative lack of
precision in our estimates. Compared with GDP, we are only
able to provide a relatively coarse estimate of changes in con-
sumer surplus given our sample size. Although the median WTA
is robust to random noise in the data (see SI Appendix for sen-
sitivity analyses of the effect of random noise and sample size on
media WTA), the overall demand schedule, including very high
or low values, is not: A small fraction of consumers with extreme
valuations can have undue influence. In contrast, focusing only
on the median valuations, while much more robust to noise,
limits the application of the SBDC approach to those goods that
are used by at least 50% of the population or requires targeting a
sample of users of these goods (in SI Appendix we also explore
other key percentiles such as the 25th and 75th percentiles).
Future work should use more massive sample sizes to narrow the
CI of the WTA estimates. Moreover, before being able to derive
surplus measures along the overall demand curve, we need fur-
ther evidence to confirm that the error variance in the data re-
mains consistent over time and therefore cancels out when
calculating annual changes. Another limitation of our study is
that it is biased toward people using the internet. The choice
experiments are only accessible online, and therefore people not
using the internet at all [around 11% of the US population (24)]
are excluded.
Despite their limitations, the choice experiments we conduct
are at least attempting to directly measure a concept that we
know is not correctly measured by other official data. In short,
we believe it is better to be approximately correct than precisely
Materials and Methods
See SI Appendix for a detailed description of all materials and methods used
within this study as well as additional discussion on why GDP and welfare
need not be correlated and summaries of previous research on measuring
welfare gains from digital goods. Our studies were approved by the Insti-
tutional Review Boards (IRB) of the Massachusetts Institute of Technology
and the University of Groningen. For the short hypothetical SBDC experi-
ments run on Google Surveys, informed consent was not required as the IRB
determined that these studies were exempt. For all of the remaining studies,
including the incentive-compatible SBDC experiments, informed consent
was obtained on the first page of the study.
ACKNOWLEDGMENTS. We thank Susanto Basu, Carol Corrado, Erwin Diewert,
Kevin Fox, Jana Gallus, Robert Hall, John Hauser, Leonard Nakamura, Hal
Varian, and participants of the Conference on Research on Income and
Wealth (2016) and the American Economic Association annual meeting
(2018) for helpful comments. We thank the Massachusetts Institute of
Technology Initiative on the Digital Economy, via a grant from the Markle
Foundation, for generous funding.
1. Federal Reserve Bank of Boston (2003) How do we measure standard of living?
Available at
pdf. Accessed January 19, 2019.
2. Feldstein M (2017) Underestimating the real growth of GDP, personal income, and
productivity. J Econ Perspect 31:145164.
3. Stiglitz J, Sen A, Fitoussi JP (2009) Report by the commission on the measurement
of economic performance and social progress. Available at
eurostat/documents/118025/118123/Fitoussi+Commission+report. Accessed January 19,
4. Bean C (2016) Independent review of UK economic statistics (Cabinet Office, London).
Available at
uk-economic-statistics-final-report. Accessed January 19, 2019.
5. Krueger AB, Stone AA (2014) Psychology and economics. Progress in measuring
subjective well-being. Science 346:4243.
6. den Haan W, Ellison M, Ilzetzki E, McMahon M (2017) Happiness and wellbeing as
objectives of macroeconomic policy: Views of economists. Available at https://v oxeu.
7. Dynan K, Sheiner L (2018) GDP as a measure of economic well-being. Available at Accessed
January 19, 2019.
8. Center for the Digital Future at USC Annenberg (2019) The 2018 digital future report.
Available at
Future-Report.pdf. Accessed January 8, 2019.
9. Facebook (2019) Facebook company information. Available at https://newsroom.fb.
com/company-info/. Accessed January 8, 2019.
10. Constine J (2018) WhatsApp hits 1.5 billion monthly users. $19B? Not so bad. Available
bad/. Accessed January 8, 2019.
11. Chen Y, Jeon GY, Kim Y-M (2014) A day without a search engine: An experimental
study of online and offline searches. Exp Econ 17:512536.
12. Giles J (2005) Internet encyclopaedias go head to head. Nature 438:900901.
13. Brynjolfsson E, Oh J (2012) The attention economy: Measuring the value of free
digital services on the Internet. Proceedings of the 33rd International Conference on
Information Systems. Available at content.cgi?article=1045&
14. Nordhaus WD (2004) Schumpeterian profits in the American economy: Theory and
measurement. NBER Working Paper No. 10433 (National Bureau of Economic Research,
Cambridge, MA).
15. Groshen EL, Moyer BC, Aizcorbe AM, Bradley R, Friedman DM (2017) How government
statistics adjust for potential biases from quality change and new goods in an age of
digital technologies: A view from the trenches. JEconPerspect31:187210.
Brynjolfsson et al. PNAS Latest Articles
16. Spence M, Owen B (1977) Television programming, monopolistic competition, and
welfare. Q J Econ 91:103126.
17. Carson RT, Groves T, List JA (2014) Consequentiality: A theoretical and experimental
exploration of a single binary choice. J Assoc Environ Resour Econ 1:171207.
18. Becker GM, DeGroot MH, Marschak J (1964) Measuring utility by a single-response
sequential method. Behav Sci 9:226232.
19. Flynn TN, Louviere JJ, Peters TJ, Coast J (2007) BestWorst scaling: What it can do for
health care research and how to do it. J Health Econ 26:171189.
20. Sunstein C (February 27, 2019), Valuing Facebook. Behavioural Public Policy, 10.1017/
21. Byrne D, Corrado C (2018) Accoun ting for innovation in consumer digital services: Impli-
cations for economic growth and consumer welfare. Available at
programme-escoe-conference-on-economic-measurement-2018-2/. Accessed January 8, 2019.
22. Allcott H, Gentzkow M (2017) Social media and fake news in the 2016 election. J Econ
Perspect 31:211236.
23. Kraut R, Burke M (2015) Internet use and psychological well-being: Effects of activity
and audience. Commun ACM 58:94100.
24. Anderson M, Perrin A, Jiang J (2018) 11% of Americans dont use the internet. Who
are they? Available at
dont-use-the-internet-who-are-they/. Accessed January 8, 2019.
| Brynjolfsson et al.
... Given this rapid adoption and usage of social media platforms, it is essential to study the impact of social media on the well-being of users. Brynjolfsson et al. [3] find that digital technologies, including social media, generate a large amount of consumer surplus. More specifically, they conduct incentive compatible choice experiments to measure the consumer surplus generated by Facebook and find that the median US Facebook user obtains around $48/month of value from using Facebook in 2017 as measured from their willingness to accept to give up access to Facebook for a month. ...
... It is interesting to notice that while social media generates large amount of consumer surplus [3], it doesn't seem to affect the subjective well-being of users. Future research can explore this wedge between consumer surplus and subjective well-being and see whether they are correlated for some products and uncorrelated for others. ...
Full-text available
Recent research has shown that social media services create large consumer surplus. Despite their positive impact on economic welfare, concerns are raised about the negative association between social media usage and well-being or performance. However, causal empirical evidence is still scarce. To address this research gap, we conduct a randomized controlled trial among students in which we track participants’ daily digital activities over the course of three quarters of an academic year. In the experiment, we randomly allocate half of the sample to a treatment condition in which social media usage (Facebook, Instagram, and Snapchat) is restricted to a maximum of 10 minutes per day. We find that participants in the treatment group substitute social media for instant messaging and do not decrease their total time spent on digital devices. Contrary to findings from previous correlational studies, we do not find any significant impact of social media usage as it was defined in our study on well-being and academic success. Our results also suggest that antitrust authorities should consider instant messaging and social media services as direct competitors before approving acquisitions.
... 13 As an additional and novel measure, we also estimate incidence as the ratio of consumer surplus losses from carbon pricing policies against total consumer surplus from air travel. Total consumer welfare is rarely measured because of the challenge of constructing full demand curvessee Hausman (1999), Cohen et al. (2016), Brynjolfsson et al. (2019) for a few exceptions. Since we produce full demand curves for each income quintile, we are able to estimate the total consumer surplus from air travel for each quintile. ...
Full-text available
This paper investigates the trade-offs between progressivity and effectiveness for a carbon tax versus an ‘excessive consumption’ levy. To do this, we compare the distribution of consumer welfare impacts and environmental effectiveness of an air travel carbon tax and a frequent flyer levy. Results show that both policies have the potential to achieve substantial carbon mitigation with minimal impacts on consumer welfare. Nevertheless, compared with a carbon tax, a frequent flyer levy is more progressive and effective at reducing emissions – thus, there is no trade-off between progressivity and effectiveness by using an excessive consumption levy to mitigate air travel emissions. Furthermore, considering the pronounced growth in demand projected for air travel over the next 30 years, results show the frequent flyer levy will remain more progressive and effective over time. Although further research is needed to assess the trade-offs on the supply-side (e.g., protection of regular customers, dynamic efficiency) and related to implementation (e.g., data privacy, the role for revenue recycling), such an excessive consumption levy has the potential to be an equitable, effective and politically acceptable environmental policy for curbing carbon dioxide emissions. This is relevant not only for air travel but for other forms of consumption in which the affluent are responsible for a large share of demand and associated carbon emissions.
... Our work aims to raise the bar for user studies for hardware design by introducing incentive compatible methodologies and mechanisms. In particular, take inspiration from previous incentive compatible studies, and in particular from Brynjolfsson, Collis, and Eggers, who conducted incentive compatible experiments to determine the value created by free online digital goods (such as access to web searches, online maps, social media, etc.) [4]; this work produces incentive compatible results via mechanisms such as Becker-DeGroot-Marschak (BDM) lotteries [3], Best-Worst Scaling (BWS) [15],and single discrete binary choice (SDBC) experiments [5]. We use the SDBC mechanism for achieving incentive compatibility in our own work but note that other mechanisms may be useful for future experiments on consumers' willingness to pay for system features like performance or security. ...
Full-text available
Architects and systems designers artfully balance multiple competing design constraints during the design process but are unable to translate between system metrics and end user experience. This work presents three methodologies to fill in this gap. The first is an incentive-compatible methodology that determines a "ground truth" measurement of users' value of speed in terms of US dollars, and find that users would accept a performance losses of 10%, 20%, and 30% to their personal computer in exchange for \$2.27, \$4.07, and \$4.43 per day, respectively. However, while highly accurate the methodology is a painstaking process and does not scale with large numbers of participants. To allow for scalability, we introduce a second methodology -- a lab-based simulation experiment -- which finds that users would accept a permanent performance loss of 10%, 20%, and 30% to their personal computer in exchange for \$127, \$169, and \$823, respectively. Finally, to allow for even greater scalability, we introduce a third methodology -- a survey -- and observe that the lack of incentive compatibility and the lack of hands-on experience with throttled device performance skews the results significantly, thus demonstrating the need for lab-based or incentive compatible study designs. By quantifying the tradeoff between user satisfaction and performance, we enable architects and systems designers to make more nuanced tradeoffs between design requirements.
Technical Report
Full-text available
The Scoping Culture and Heritage Capital study was commissioned jointly in November 2021 by the Arts and Humanities Research Council (AHRC) and Department for Digital, Culture, Media and Sport (DCMS). The study was led by Dr Patrycja Kaszynska, Senior Research Fellow at UAL , in partnership with cultural sector partners and policy makers, and collaborating with a team of researchers spanning arts and humanities, heritage science and economics: Dr Sadie Watson and Dr Emma Dwyer from Museum of London Archaeology (MOLA); Prof Diane Coyle, University of Cambridge; Prof Patrizia Riganti and Dr Yang Wang University of Glasgow, Dr Ricky Lawton, Simetrica-Jacobs and Dr Mafalda Dâmaso (UAL). The findings of the scoping study are that the introduction of the CHC framework presents significant opportunities from the point of view of valuing the arts, culture and heritage, as well as policy decision-making as such. However, the scoping exercise shows that developing, operationalising and implementing this framework requires sustained research attention, methods refinement and, crucially, capacity- and capability-building across disciplines and sectors. This is not least because the value of arts, culture and heritage as conceived through the CHC framework is an inter- and trans-disciplinary concept. The scoping study is accompanied by a AHRC/DCMS funding call for new research informed by the project’s recommendations.
The share of artificial intelligence (AI) jobs in total job postings has increased from 0.20% to nearly 1% between 2010 and 2019, but there is significant heterogeneity across cities in the United States (US). Using new data on AI job postings across 343 US cities, combined with data on subjective well-being and economic activity, we uncover the central role that service-based cities play to translate the benefits of AI job growth to subjective well-being. We find that cities with higher growth in AI job postings witnessed higher economic growth. The relationship between AI job growth and economic growth is driven by cities that had a higher concentration of modern (or professional) services. AI job growth also leads to an increase in the state of well-being. The transmission channel of AI job growth to increased subjective well-being is explained by the positive relationship between AI jobs and economic growth. These results are consistent with models of structural transformation where technological change leads to improvements in well-being through improvements in economic activity. Our results suggest that AI-driven economic growth, while still in the early days, could also raise overall well-being and social welfare, especially when the pre-existing industrial structure had a higher concentration of modern (or professional) services.
Technical Report
Full-text available
Was ist der Wert von digitalen Daten? Die Ermittlung des Werts von Daten ist von entscheidender Bedeutung, wenn es darum geht, kleine Unternehmen zu einer besseren Nutzung ihres Datenbestands für ihre Wertschöpfungsprozesse anzuleiten. Die Quantifizierung des Werts von Daten wurde durch ein breites Spektrum von Methoden gelöst, um die besonderen wirtschaftlichen Eigenschaften von Daten in Betracht zu nehmen. Wir zeigen in dieser umfangreichen Literaturstudie, dass noch keine einheitliche Bewertungsmethode entwickelt wurde, aber wir konnten Empfehlungen zur Auswahl von Verfahren für Unternehmen sammeln, die bereit sind, die Bewertung ihrer Daten durchzuführen. Wir haben ein Bewertungsverfahren entwickelt, um sie anzuleiten, wobei wir versucht haben, einen breiten Anwendungsbereich beizubehalten, so dass es in den häufigsten Fällen eingesetzt werden kann.
Smartphones can lower the disutility of waiting by increasing productivity and making time pass more pleasantly. We elicit the compensation required by subjects to wait for 30 minutes, alone in an empty room, under four different conditions that varied access to the subject’s smartphone. Compared to the treatment where subjects had full use of their phone, we find that they required 24% percent more to wait with the audio features of the phone remaining but the phone physically locked away, 48% percent more to wait with only an FM radio, and 79% percent more to wait in a quiet room. We find little correlation between a subject’s wages and her offers, emphasizing the importance of heterogeneity in the value of time that is based on context rather than income.
Full-text available
A key economic indicator is real output. To get this right, we need to measure accurately both the value of nominal GDP (done by Bureau of Economic Analaysis) and key price indexes (done mostly by Bureau of Labor Statisticcs). All of us have worked on these measurements while at the BLS and the BEA. In this article, we explore some of the thorny statistical and conceptual issues related to measuring a dynamic economy. An often-stated concern is that the national economic accounts miss some of the value of some goods and services arising from the growing digital economy. We agree that measurement problems related to quality changes and new goods have likely caused growth of real output and productivity to be understated. Nevertheless, these measurement issues are far from new, and, based on the magnitude and timing of recent changes, we conclude that it is unlikely that they can account for the pattern of slower growth in recent years. First we discuss how the Bureau of Labor Statistics currently adjusts price indexes to reduce the bias from quality changes and the introduction of new goods, along with some alternative methods that have been proposed. We then present estimates of the extent of remaining bias in real GDP growth that stem from potential biases in growth of consumption and investment. And we take a look at potential biases that could result from challenges in measuring nominal GDP, including those involving the digital economy. Finally, we review ongoing work at BLS and BEA to reduce potential biases and further improve measurement.
Full-text available
Researchers, using contingent valuation (CV) to value changes in nonmarket goods, typically believe respondents always answer questions truthfully or they answer truthfully only when it is in their interest to do so. The second position, while consistent with economic theory, implies that interpreting survey responses depends critically on the incentive structure provided. We derive simple tests capable of distinguishing the two views. Our theoretical model for examining the incentive structure of a single binary choice relaxes the usual expected utility assumption. We test our theory using a field experiment involving voting to provide a public good. Experimental results are consistent theoretical predictions and cast doubt on the relevance of a large experimental literature using inconsequential questions and non-incentive-compatible mechanisms to make inferences about CV. The framework put forth should help in understanding the role played by theoretical conditions for preference elicitation and lend insight into the hypothetical bias literature.
In recent years, there has been a great deal of discussion of the welfare effects of digital goods, including social media. A national survey, designed to monetize the benefits of a variety of social media platforms (including Facebook, Twitter, YouTube and Instagram), found a massive disparity between willingness to pay (WTP) and willingness to accept (WTA). The sheer magnitude of this disparity reflects a ‘superendowment effect’. Social media may be Wasting Time Goods – goods on which people spend time, but for which they are not, on reflection, willing to pay much (if anything). It is also possible that in the context of the WTP question, people are giving protest answers, signaling their intense opposition to being asked to pay for something that they had formerly enjoyed for free. Their answers may be expressive, rather than reflective of actual welfare effects. At the same time, the WTA measure may also be expressive, a different form of protest, telling us little about the actual effects of social media on people's lives and experiences. It may greatly overstate those effects. In this context, there may well be a sharp disparity between conventional economic measures and actual effects on experienced well-being.
Following the 2016 US presidential election, many have expressed concern about the effects of false stories ("fake news"), circulated largely through social media. We discuss the economics of fake news and present new data on its consumption prior to the election. Drawing on web browsing data, archives of fact-checking websites, and results from a new online survey, we find: 1) social media was an important but not dominant source of election news, with 14 percent of Americans calling social media their "most important" source; 2) of the known false news stories that appeared in the three months before the election, those favoring Trump were shared a total of 30 million times on Facebook, while those favoring Clinton were shared 8 million times; 3) the average American adult saw on the order of one or perhaps several fake news stories in the months around the election, with just over half of those who recalled seeing them believing them; and 4) people are much more likely to believe stories that favor their preferred candidate, especially if they have ideologically segregated social media networks.
The problems involved in estimating real output that I discuss in this paper cause the official government statistics to underestimate of the rates of growth of real GDP, real personal income, and productivity. That underestimation is important not just to economists trying to understand where the economy is going but also to the broader public and to the political system. The understatement of real growth reflects the enormous difficulty of dealing with quality change and the even greater difficulty of measuring the value created by the introduction of new goods and services. Despite the vast amount of attention that has been devoted to this subject in the economic literature and by the government agencies, there remains insufficient understanding of just how imperfect the official estimates actually are. It is important for economists to recognize the limits of our knowledge and to adjust public statements and policies to what we can know. This paper is not about the recent slowdown in measured productivity but that subject is discussed briefly.
Over the past decade, there has been an explosion of digital services on the Internet, from Google and Wikipedia to Facebook and YouTube. However, the value of these innovations is difficult to quantify, because consumers pay nothing to use them. We develop a new framework to measure the value of free services using the insight that even when people do not pay cash, they must still pay "attention," or time. Using our model, we estimate the increase in consumer surplus created by free internet services to be over $100 billion per year in the U.S. alone. Our analysis implies that most of welfare gain from digital services on the Internet would be overlooked by traditional approaches that rely only on the direct expenditures of money. Considering the time spent on consumption, as we do, makes it possible to assess the full value of these digital innovations.
The connection between online communication and psychological well-being depends on whom you are communicating with.
With the advent of the Web and search engines, online search has become a common method of obtaining information. The question arises as to how much time people save by using search engines for their information needs, and the extent to which online search affects search experiences and outcomes. Using a random sample of queries from a major search engine, we conduct an experiment to compare online and offline search experiences and outcomes. We find that participants are significantly more likely to find an answer on the Web. Restricting to the set of queries which participants find answers in both treatments, the average search time is 22 minutes offline, and 7 minute online. While library sources are judged to be significantly more trustworthy and authoritative than the corresponding web sources, web sources are judged to be significantly more relevant and more likely to contain enough information to answer the question. Balancing all factors, the overall source quality is not significantly different between the two treatments. Lastly, post-search questionnaires reveal that online search is more enjoyable than offline search.
Progress in science requires new tools for measuring phenomena previously believed unmeasurable, as well as conceptual frameworks for interpreting such measurements. There has been much progress on both fronts in the measurement of subjective well-being (SWB), which “refers to how people experience and evaluate their lives and specific domains and activities in their lives” ( 1 ). In 2009, the Sarkozy Commission recommended adding SWB measures as supplements to existing indicators of societal progress such as gross domestic product (GDP). In light of subsequent activity by governments and international organizations, we summarize several important advances and highlight key remaining methodological challenges that must be addressed to develop a credible national indicator of SWB and to incorporate SWB into official statistics and policy decisions.