No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness
Abstract
To measure the effects of advertising, marketers must know how consumers would behave had they not seen the ads. The authors develop a methodology they call ‘Ghost Ads,’ which facilitates this comparison by identifying the control-group counterparts of the exposed consumers in a randomized experiment. The authors show that, relative to Public Service Announcement (PSA) and Intent-to-Treat A/B tests, ‘Ghost Ads’ can reduce the cost of experimentation, improve measurement precision, deliver the relevant strategic baseline, and work with modern ad platforms that optimize ad delivery in real-time. The authors also describe a variant ‘Predicted Ghost Ad’ methodology that is compatible with online display advertising platforms; their implementation records more than 100 million predicted ghost ads per day. The authors demonstrate the methodology with an online retailer's display retargeting campaign. They show novel evidence that retargeting can work as the ads lifted website visits by 17.2% and purchases by 10.5%. Compared to Intent-to-Treat or PSA experiments, advertisers can measure ad lift just as precisely while spending at least an order of magnitude less.
... However, is not , in general, proportional to the potential ad stock from (t) z i winning because the probability of winning the auction can be very heterogeneous across each bid opportunity j and user i . This implies a simple adjustment to the model to unambiguously improve its predictive power, similar to "predicted Ghost Ads" (Johnson, Lewis, and Nubbemeyer 2017), by using a predicted win probability for each auction, , given the ...
... In order to restore conditional independence, we construct the analogous exogenous W regressors via "ghost bids" or "predicted ghost ads" (Johnson, Lewis, and Nubbemeyer 2017). Here, these continuoustime "ghost bid stock" are defined based on the user's context ; we ...
... Then, exogeneity of is violated for (t) ξ i subsequent bids if our bidding algorithm depends on past exposures, either explicitly via frequency capping or implicitly via changes in any user behavior caused by the advertising that changes future model bids (e.g., a form of endogeneity referred to as "covariate shift" in the machine learning literature). Johnson, Lewis, and Nubbemeyer (2017) provide examples of this bias in their implementation of predicted ghost ads. This weakness of predicted ghost ads in userlevel randomization should discourage the use of userlevel randomization as the primary source of exogeneity for estimating statistically precise incrementality bidding or attribution models. ...
The causal effect of showing an ad to a potential customer versus not, commonly referred to as "incrementality", is the fundamental question of advertising effectiveness. In digital advertising three major puzzle pieces are central to rigorously quantifying advertising incrementality: ad buying/bidding/pricing, attribution, and experimentation. Building on the foundations of machine learning and causal econometrics, we propose a methodology that unifies these three concepts into a computationally viable model of both bidding and attribution which spans the randomization, training, cross validation, scoring, and conversion attribution of advertising's causal effects. Implementation of this approach is likely to secure a significant improvement in the return on investment of advertising.
... Following the early work on why marketers may want to use targeting for communication efforts, a literature stream on the efficiency of customer targeting and customized content related to digital marketing emerged (Bucklin & Sismeiro, 2003;Ansari & Mela, 2003;Park & Fader, 2004;Manchanda et al., 2006;Hauser et al., 2009). This body of work expanded subsequently into all aspects of technologydriven targeting, such as contextual targeting (Goldfarb & Tucker, 2011), behavioral targeting (Summers et al., 2016), keyword targeting (Li et al., 2016), mobile targeting (Luo et al., 2014;Andrews et al., 2016;Chenet al., 2017) and re-targeting (Lambrecht & Tucker, 2013;Johnson et al., 2017). ...
... It should be noted that the integration of the measurement company with publishers means we can replicate how ad serving process would look like for a real publisher network without actually serving an ad. This replicates some of the motivation behind the Ghost Ads methodology (Johnson et al., 2017), which similarly was focused on saving media costs. All survey answers were collected anonymously. ...
Often marketers face the challenge of how to communicate best with the customers who have the right responsibilities, influence or purchasing power, especially in business-to-business (B2B) settings. For example, B2B marketers selling software and IT need to identify IT decision-makers (ITDMs) within organizations. The modern digital environment in theory allows marketers to target individuals in organizations through specifically designed third-party audience segments based on deterministic prospect lists or probabilistic inference. However, in this paper we show that in our context, such ‘off-the-shelf’ segments perform no better at reaching the right person than random prospecting. We present evidence that even deterministic attribute information is flawed for ITDM identification, and that the poor campaign results can be partly linked to incorrect assignment of established prospect profiles to online identifiers. We then use access to our publisher network data to investigate what would happen if the advertiser had used first-party data that are less susceptible to the identified issues. We demonstrate that first-party demographics or contextual information allows advertisers and publishers to outperform both third-party ITDM audience segments and random prospecting. Our findings have implications for understanding the shift in digital advertising away from third-party cookie tracking, and how to execute digital marketing in the context of broad privacy regulation.
... More recently, Gordon et al. (2019), and Gordon et al. (2022) have shown that even highquality data and machine learning models may be insufficient to reliably recover advertising effects using the data typically collected by advertising platforms. A key innovation in this literature came from Johnson et al. (2017a), who describe a way to increase the power of ad experiments through identifying the subset of users in the control group who would have received treatment. Experiments have also allowed researchers to examine complex theories of advertising effects (Sahni, 2015;Sahni and Nair, 2020). ...
... RCTs also require pre-planning before they are run and cannot be implemented after the fact. Also, RCTs impose opportunity costs for advertising platforms since common approaches such as "Ghost Ads" Johnson et al. (2017a) require punting the highest-ranked ad for the control group and replacing it with the second-highest ad. This decreases the price an advertising platform can charge for ad impressions in the control group. ...
We present a novel approach to causal measurement for advertising, namely to use exogenous variation in advertising exposure (RCTs) for a subset of ad campaigns to build a model that can predict the causal effect of ad campaigns that were run without RCTs. This approach -- Predictive Incrementality by Experimentation (PIE) -- frames the task of estimating the causal effect of an ad campaign as a prediction problem, with the unit of observation being an RCT itself. In contrast, traditional causal inference approaches with observational data seek to adjust covariate imbalance at the user level. A key insight is to use post-campaign features, such as last-click conversion counts, that do not require an RCT, as features in our predictive model. We find that our PIE model recovers RCT-derived incremental conversions per dollar (ICPD) much better than the program evaluation approaches analyzed in Gordon et al. (forthcoming). The prediction errors from the best PIE model are 48%, 42%, and 62% of the RCT-based average ICPD for upper-, mid-, and lower-funnel conversion outcomes, respectively. In contrast, across the same data, the average prediction error of stratified propensity score matching exceeds 491%, and that of double/debiased machine learning exceeds 2,904%. Using a decision-making framework inspired by industry, we show that PIE leads to different decisions compared to RCTs for only 6% of upper-funnel, 7% of mid-funnel, and 13% of lower-funnel outcomes. We conclude that PIE could enable advertising platforms to scale causal ad measurement by extrapolating from a limited number of RCTs to a large set of non-experimental ad campaigns.
... Secondly, although the study design is similar to a randomized field experiment, there are differences related to the placement algorithm on Facebook which are out of the control of researchers [12,13]. This complicates a causal interpretation of the effects, as endogeneity cannot be ruled out completely. ...
Background
Many high-income countries are grappling with severe labour shortages in the healthcare sector. Refugees and recent migrants present a potential pool for staff recruitment due to their higher unemployment rates, younger age, and lower average educational attainment compared to the host society's labour force. Despite this, refugees and recent migrants, often possessing limited language skills in the destination country, are frequently excluded from traditional recruitment campaigns conducted solely in the host country’s language. Even those with intermediate language skills may feel excluded, as destination-country language advertisements are perceived as targeting only native speakers. This study experimentally assesses the effectiveness of a recruitment campaign for nursing positions in a German care facility, specifically targeting Arabic and Ukrainian speakers through Facebook advertisements.
Methods
We employ an experimental design (AB test) approximating a randomized controlled trial, utilizing Facebook as the delivery platform. We compare job advertisements for nursing positions in the native languages of Arabic and Ukrainian speakers (treatment) with the same advertisements displayed in German (control) for the same target group in the context of a real recruitment campaign for nursing jobs in Berlin, Germany. Our evaluation includes comparing link click rates, visits to the recruitment website, initiated applications, and completed applications, along with the unit cost of these indicators. We assess statistical significance in group differences using the Chi-squared test.
Results
We find that recruitment efforts in the origin language were 5.6 times (Arabic speakers) and 1.9 times (Ukrainian speakers) more effective in initiating nursing job applications compared to the standard model of German-only advertisements among recent migrants and refugees. Overall, targeting refugees and recent migrants was 2.4 (Ukrainians) and 10.8 (Arabic) times cheaper than targeting the reference group of German speakers indicating higher interest among these groups.
Conclusions
The results underscore the substantial benefits for employers in utilizing targeted recruitment via social media aimed at foreign-language communities within the country. This strategy, which is low-cost and low effort compared to recruiting abroad or investing in digitalization, has the potential for broad applicability in numerous high-income countries with sizable migrant communities. Increased employment rates among underemployed refugee and migrant communities, in turn, contribute to reducing poverty, social exclusion, public expenditure, and foster greater acceptance of newcomers within the receiving society.
... Johnson, G. A. et al. developed a phantom advertising method to measure the effectiveness of advertising and marketing and validated the method through an example study. This phantom advertising method greatly reduces the marketing cost [20]. Sun, B. et al. combined a variety of analytical tools and methods, such as cluster analysis, sentiment analysis, etc., to explore the core demands of automobile consumers' car purchases, including gas mileage, cost-effective face value, and six other aspects, which is of positive significance for the enterprise's automotive precision marketing program development [21]. ...
Precision marketing emerges as a pivotal mechanism to foster market development, enhancing the market economy and propelling internal economic growth. This study collates data pertinent to marketing activities, standardizes the raw datasets to compute descriptive statistics such as the mean and variance, and constructs a correlation coefficient matrix under specified conditions. Through factor analysis, the structure of this correlation matrix is meticulously examined. Factor loadings are employed to elucidate the relationships between factors and variables, thereby establishing a link between the precision of marketing endeavors and consumer attribute variables for further analytical probing. To assess the impact of implemented marketing strategies on consumer response behaviors, this research develops a precision marketing model using the Uplift algorithm. This model innovatively addresses the challenge of individual causal effects—which are realistically unresolvable—by transforming it into estimating the conditional average causal utility derivable from observed data. The factor analysis feasibility test yields a Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy at 0.792, and Bartlett's test of sphericity attains a significance level of 0.000, indicating robust factorability. Subsequent tests on variance and principal factor extraction reveal that five variables—such as income, gender, and age—are common factors across the datasets analyzed. Application of the Uplift model to the MineThatData and MegaFon datasets further substantiates the efficacy of the proposed marketing model. Particularly, the results from the MegaFon dataset validate the comprehensive applicability of this model, demonstrating its effectiveness in real-world scenarios.
... For instance, Google's "ghost ads" approach creates a holdout group that is a representative subset of consumers who were targeted with and just about to be exposed to the focal ad, but then were randomly not shown the ad. Therefore, the mix of users in the holdout is the same as the mix in group A that was exposed to the focal ad, supporting an analysis of the average treatment effect on the treated (Johnson, Lewis, and Nubbemeyer 2017). Meta's "conversion lift" approach, on the other hand, forms a holdout with the same mix as all exposed and unexposed consumers in group A, supporting an intent-to-treat analysis, where the platform "intent" is targeting a user to be exposed regardless of whether they are available for exposure (Gordon et al. 2019;Gordon, Moakler, and Zettelmeyer 2023). ...
Digital advertising platforms have emerged as a widely utilized data source in consumer research; yet, the interpretation of such data remains a source of confusion for many researchers. This article aims to address this issue by offering a comprehensive and accessible review of four prominent data collection methods proposed in the marketing literature: “informal studies,” “multiple-ad studies without holdout,” “single-ad studies with holdout,” and “multiple-ad studies with holdout.” By outlining the strengths and limitations of each method, we aim to enhance understanding regarding the inferences that can and cannot be drawn from the collected data. Furthermore, we present seven recommendations to effectively leverage these tools for programmatic consumer research. These recommendations provide guidance on how to use these tools to obtain causal and non-causal evidence for the effects of marketing interventions, and the associated psychological processes, in a digital environment regulated by targeting algorithms. We also give recommendations for how to describe the testing tools and the data they generate and urge platforms to be more transparent on how these tools work.
... The existing research has acknowledged the gap in understanding the post-purchase effects of retargeting ads. Johnson et al. (2017) stated that if retargeting ads lead to reactance in consumers, it can undermine the effectiveness of advertising. Therefore, as discussed by Baek and Morimoto (2012), it is essential to acknowledge that the in-fluence of these ads goes beyond their role in driving initial purchases. ...
This study aims to explore the complex effects of post-purchase retargeting ads on consumer behavior, with a focus on expectation confirmation, satisfaction, and repurchase intentions. Additionally, it examines the influence of time spent online on these effects. Anchored in expectation confirmation theory (ECT), the study analyzes responses from 396 Saudi Arabian e-tourism customers who encountered competitive retargeting ads after purchasing an e-tourism package. The analysis employs partial least squares structural equation modeling (PLS-SEM) and multigroup analysis (MGA) to test the hypotheses. A notable finding is the direct negative impact of retargeting ads on expectation confirmation: increased exposure to such ads post-purchase seems to diminish the perception that initial expectations of the product or service are being met. The negative effect of these ads also indirectly influences satisfaction and repurchase intentions. Furthermore, the MGA results indicate variations in this negative impact based on the time spent online. Specifically, the more time consumers spend online, the stronger the negative impact, leading to a significant decrease in satisfaction and repurchase intentions. These insights reveal the complex nature of post-purchase retargeting ads and underscore the importance of accounting for consumers’ online behavior. They offer valuable direction for marketers to refine retargeting strategies to better resonate with consumer expectations.
... uncovering the incremental impact of banner advertising on website visits and purchases is however difficult (e.g., with experiments using ghost ads) and typically need large-scale field experiment (see, e.g., Johnson, Lewis, and Nubbemeyer 2017 ² This gives the sample size. In total, 599 participants evaluated six sites and asked about eight product categories of interest, resulting in 3,594 site evaluations and 4,792 potential brand mentions. ...
A growing group of consumers uses ad-blocking software, preventing advertisers from reaching them and resulting in a loss of ad revenue for publishers. Ways to resolve this issue include blocking these users, disguising ads, or paying the developer of the ad blocker so that ads will not be blocked. The question is to what extent these solutions are effective and desired. This study uses an experimental setup followed by an extensive survey to answer this question. The findings show that, when banner ads are forced on ad blocker users, these users (vs. ad blocker nonusers) spend 10%-20% less time on the web page, evaluate the website as worse, and pay less attention to the banners, while the ads are 190% more effective for ad blocker nonusers. Thus, ad blocking serves as a self-filtering mechanism that filters out consumers who are less responsive to advertising. Ad blockers thus help advertisers target the right consumers and increase the value of the remaining ad slots for publishers. Moreover, ad blocker users are more likely to pay for ad-free content, offering publishers an alternative business model for these consumers.
... The usage of field experiments for determination of ad effectiveness has subsequently proliferated with studies done on Facebook (Gordon et al., 2019) and Google (Johnson, Lewis, and Nubbemeyer 2017) by creating randomized control groups. A field experiment by Sahni (2016) showed that digital ads for one restaurant led to an increase in sales at a competitor restaurant that offered similar cuisine. ...
... Additionally, Neyman-Pearson testing does not allow marketing professions to incorporate detailed expert knowledge. For example, Frequentist vs Bayesian A/B testing online advertising campaigns often yield minuscule increases in conversion rates because of poor reliability of statistical decisions (Johnson, Lewis, & Nubbemeyer, 2017). In contrast, the Bayesian framework is conceptually straightforward, incorporates expert knowledge and results in more informed statistical analyses (Lindley, 1993). ...
Purpose
This paper tests whether Bayesian A/B testing yields better decisions that traditional Neyman-Pearson hypothesis testing. It proposes a model and tests it using a large, multiyear Google Analytics (GA) dataset.
Design/methodology/approach
This paper is an empirical study. Competing A/B testing models were used to analyze a large, multiyear dataset of GA dataset for a firm that relies entirely on their website and online transactions for customer engagement and sales.
Findings
Bayesian A/B tests of the data not only yielded a clear delineation of the timing and impact of the intellectual property fraud, but calculated the loss of sales dollars, traffic and time on the firm’s website, with precise confidence limits. Frequentist A/B testing identified fraud in bounce rate at 5% significance, and bounces at 10% significance, but was unable to ascertain fraud at the standard significance cutoffs for scientific studies.
Research limitations/implications
None within the scope of the research plan.
Practical implications
Bayesian A/B tests of the data not only yielded a clear delineation of the timing and impact of the IP fraud, but calculated the loss of sales dollars, traffic and time on the firm’s website, with precise confidence limits.
Social implications
Bayesian A/B testing can derive economically meaningful statistics, whereas frequentist A/B testing only provide p -value’s whose meaning may be hard to grasp, and where misuse is widespread and has been a major topic in metascience. While misuse of p -values in scholarly articles may simply be grist for academic debate, the uncertainty surrounding the meaning of p -values in business analytics actually can cost firms money.
Originality/value
There is very little empirical research in e-commerce that uses Bayesian A/B testing. Almost all corporate testing is done via frequentist Neyman-Pearson methods.
Marketers use online advertising platforms to compare user responses to different ad content. But platforms’ experimentation tools deliver different ads to distinct and undetectably optimized mixes of users that vary across ads, even during the test. Because exposure to ads in the test is nonrandom, the estimated comparisons confound the effect of the ad content with the effect of algorithmic targeting. This means that experimenters may not be learning what they think they are learning from ad A/B tests. The authors document these “divergent delivery” patterns during an online experiment for the first time. They explain how algorithmic targeting, user heterogeneity, and data aggregation conspire to confound the magnitude, and even the sign, of ad A/B test results. Analytically, the authors extend the potential outcomes model of causal inference to treat random assignment of ads and user exposure to ads as separate experimental design elements. Managerially, the authors explain why platforms lack incentives to allow experimenters to untangle the effects of ad content from proprietary algorithmic selection of users when running A/B tests. Given that experimenters have diverse reasons for comparing user responses to ads, the authors offer tailored prescriptive guidance to experimenters based on their specific goals.
Varying search costs can lead to choice deferral and choice closure behavior, with effects of strategic pricing and retargeting.
Problem definition: Online retailing platforms are increasingly relying on personalized recommender systems to help guide consumer choice. An important but understudied question in such settings is how many products to include in a recommendation set. In this work, we study how the number of recommended products influences consumers’ search and purchase behavior in an online personalized recommender system within a retargeting setting. Methodology/results: Via a field experiment involving 1.6 million consumers on an online retailing platform, we causally demonstrate that consumers’ likelihood of purchasing any product from the recommendation set first increases then decreases as the number of recommended products increases. Importantly, as much as 64% of the decrease in purchase probability (i.e., the choice overload effect) can be attributed to a decrease in consumers’ likelihood of starting a search (i.e., clicking on any recommended product). We discuss the possible behavioral mechanisms driving these results and analyze how these effects could be heterogeneous across different product categories, price ranges, and timing. Managerial implications: This work presents real-world experimental evidence for the choice overload effect in online retailing platforms, highlights the important role of consumer search behavior in driving this effect, and sheds light on when and how limiting the number of options in a recommender system may be beneficial to online retailers.
Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.0659 .
This paper studies the impact of platform endorsement on consumers’ search and purchase behavior, analyzing how it affects endorsed and unendorsed items.
We run a large-scale field experiment at Meta to estimate how valuable offsite tracking data are for digital ad effectiveness.
This paper shows the existence of bias in the estimation of ad effectiveness created by concurrent online experimentation by competing advertisers and proposes a framework to account and correct for it.
The study of experimental design offers tremendous benefits for answering causal questions across a wide range of applications, including agricultural experiments, clinical trials, industrial experiments, social experiments, and digital experiments. Although valuable in such applications, the costs of experiments often drive experimenters to seek more efficient designs. Recently, experimenters have started to examine such efficiency questions from an optimization perspective, as experimental design problems are fundamentally decision-making problems. This perspective offers a lot of flexibility in leveraging various existing optimization tools to study experimental design problems. This manuscript thus aims to examine the foundations of experimental design problems in the context of causal inference as viewed through an optimization lens.
This paper introduces a multiarmed bandit approach to estimate advertising effectiveness and manage experimentation costs in real-time bidding auctions concurrently.
Multi-product ads (MPAs) allow an ad platform to display multiple products within a single ad unit. Unlike a single-product display ad, similar ads from the same product category may appear in MPAs and induce consumer satiation. Further, the simultaneous display of ads from multiple product categories can result in cross-category complementarity or substitution in consumers’ utility from ad-clicking. This study examines ad platforms’ potential benefit from incorporating consumer within-category satiation and cross-category spillover into ad-serving policies. To achieve this goal, the authors propose a full-equilibrium modeling framework to capture both the demand (i.e., consumers’ clicks) and supply sides (i.e., advertisers’ bids) of MPAs. The results from a dataset from a large display ad network reveal heterogeneity in within-category satiation across consumers and categories, and provide evidence of cross-category complementarity. The authors then demonstrate how the platform can indirectly influence the ad variety and category composition of MPAs through three counterfactuals of ad-serving policy changes. The results show that the implementation of a more privacy-preserving policy may hurt both consumer welfare and the platform’s revenue because of the reduced ad variety, which makes consumers more prone to within-category satiation and less prone to cross-category complementarity.
Does online fundraising increase charitable giving? Using the Facebook advertising tool, we implemented a natural field experiment across Germany, randomly assigning almost 8,000 postal codes to Save the Children fundraising videos or to a pure control. We studied changes in the donation revenue and frequency for Save the Children and other charities by postal code. Our geo-randomized design circumvented many difficulties inherent in studies based on click-through data, especially substitution and measurement issues. We found that (i) video fundraising increased donation revenue and frequency to Save the Children during the campaign and in the subsequent five weeks, that (ii) the campaign was profitable for the fundraiser, and that (iii) the effects were similar independent of video content and impression assignment strategy. However, we also found some crowding out of donations to other similar charities or projects. Finally, we demonstrated that click data may be an inappropriate proxy for donations and recommend that managers use careful experimental designs that can plausibly evaluate the effects of advertising on relevant outcomes.
This paper was accepted by Yan Chen, behavioral economics and decision analysis.
Funding: M. Adena acknowledges financial support from the Deutsche Forschungsgemeinschaft [Grants 417014946 and CRC TRR 190 (Project 280092119)].
Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2020.00596 .
A part of retail media wherein sponsored product listings are interleaved with organic product listings in the search results is a large and growing phenomenon. In this paper, we study the impact of displaying sponsored listings at top positions for the platform. Analyzing data from a large-scale field experiment at a leading online marketplace in India, we find nuanced results that substantially vary across product categories. In the electronics category, the sponsored listings receive fewer clicks than the organic listings that they replace. Surprisingly, this effect is reversed in the clothing category, in which the ads perform better than the displaced organic listings, suggesting that sponsored listings might help the platform identify new high-relevance products and improve search rankings for these categories. At the search level, we find that increasing the fraction of sponsored listings (by about 10% points) in the search results does not affect the performance in any product category. This implies that ads bring in additional revenue for the marketplace yet do not hurt overall consumer response (in the short run). We theorize that the variation across categories occurs because of differing degrees of information asymmetry on product relevance between the marketplace and the sellers.
The authors investigate the impact of online news content on the effectiveness of display advertising. In a randomized online experiment, participants read news articles randomly paired with brand advertisements. Leveraging non-intrusive eye-tracking technology, the authors measure individual attention to both articles and ads. The authors then measure ad recall, and participants make choices between cash and brand-specific vouchers. Heightened attention to articles results in “spillover” attention to ads on the same page which, in turn, increases both brand recall and purchase probability. The authors also consider the effect of news content type, differentiating between “hard” and “soft” news. They find that advertising next to hard news is at least as effective as advertising next to soft news. This provides evidence against the blunt implementation of “block lists” for sensitive news topics by advertisers. The authors discuss the implications of attention spillovers for firms contemplating investments in engaging news content within the digital advertising landscape.
We conduct an experiment in a simulated online bookstore to study how consumers respond to display ads with varied content.
Problem definition: Most of the display advertising inventory is sold through real-time auctions. The participants of these auctions are typically bidders (Google, Criteo, RTB House, and Trade Desk for instance) that participate on behalf of advertisers. In order to estimate the value of each display opportunity, they usually train advanced machine learning algorithms using historical data. In the labeled training set, the inputs are vectors of features representing each display opportunity, and the labels are the generated rewards. In practice, the rewards are given by the advertiser and are tied to whether a particular user converts. Consequently, the rewards are aggregated at the user level and never observed at the display level. A fundamental task that has, to the best of our knowledge, been overlooked is to account for this mismatch and split, or attribute, the rewards at the right granularity level before training a learning algorithm. We call this the label attribution problem. Methodology/results: In this paper, we develop an approach to the label attribution problem, which is both theoretically justified and practical. In particular, we develop a fixed point algorithm that allows for large-scale implementation and showcase our solution using a large-scale publicly available data set from Criteo, a large demand-side platform. We dub our approach the fixed point label attribution algorithm. Managerial implications: There is often a hidden leap of faith when transforming the advertiser’s signal into display labeling. Demand Side Platforms providers should be careful when building their machine learning pipeline and carefully solve the label attribution step.
A method is proposed to compute more precise standard errors for marketing A/B tests.
Cookies do not make purchase decisions; individuals do. Cookie-level analyses are problematic—use first-party individual data instead.
Purpose: Firms can benefit from designing sales promotion based on the analysis of consumers' physical exercise and purchase data. This study aims to study mobile exercise app data to explore how purchasing a promoted or nonpromoted product affects exercisers’ subsequent exercise and purchase behaviors.
Design/methodology/approach: Drawing from the theoretical framework of overjustification effect, this study empirically examines the effects of the purchase of promoted – monetary and nonmonetary – or nonpromoted products on relationships (1) between past and subsequent exercise behaviors and (2) between past exercise and subsequent purchase behaviors. Novel data of one million exercise activities and purchase transactions created by 7,517 mobile exercise app users were collected.
Findings: The results reveal that monetary and nonmonetary promotions have a negative effect on overall consumers’ amount of physical exercise but increase heavy exercisers’ exercise amount. In addition, nonmonetary (monetary) promotion has a positive (negative) effect on consumers’ purchase expenditure but has no moderating effect on the exercise–expenditure relationship.
Originality/value: This study provides a theoretical framework explaining how to mitigate the dark side of sales promotions while targeting right exercise consumer segments with the right promotion campaigns.
Blockchain technology is often touted as a transformative solution for improving the current over-centralized systems in our economy, where all tangible and intangible assets are traded. Driven by computer science and engineering, blockchain technology is believed to decentralize the current system with its simple yet effective property of distributed ledger technology, which guarantees data transparency that can be seen by anyone in the network. With its robustness, blockchain technology can be applied to the centralized and multi-party-controlled current advertising ecosystem. Despite its potential, there is a lack of scientific discussion linking blockchain technology and advertising. In this essay, we examine the current challenges that render the measurement of advertising effects inefficient and untrustworthy in the advertising ecosystem. By focusing on such challenges as data transparency, consumer privacy, and ad fraud, we explain how blockchain technology can enhance the advertising system, making it more trustworthy from both technical and consumer standpoints.
This paper runs a large-scale field experiment on restaurants to understand which types of businesses gain more from digital advertising.
Online display advertising is a hostile medium for field experiments. Display‐ad effects are tiny and necessitate large‐scale experiments. The experimenter has limited control because ad exposure is jointly determined by advertisers, users, algorithms, and market competition. As such, online display ads provide useful lessons for experimenters at the frontier of digital research more generally. Display‐ad experiments place renewed focus on old topics like statistical power and compliance as well as on newer issues like identity fragmentation, experimental spillovers, and incrementality optimization. In this guide, I review these challenges, best practices, and new developments.
Engaging customers online through effective customer experience design is critical, as practitioners and researchers agree that engaged customers contribute value to firms. However, in the multinational business-to-business (B2B) context—due to its complex decision-making processes, which involve various stakeholders—global marketers face challenges in their attempts to localize (versus standardize) the online experience across their regional websites to meet customer needs, which vary across cultures. Although standardization entails cost benefits, localization provides more culturally relevant customer experiences. Accordingly, to help global marketers solve this dilemma, this study examines how culture shapes the effectiveness of online customer experiences with regard to driving psychological and behavioral customer engagement in a B2B context. The study draws on survey and observed data collected from the business customers of a multinational firm who were located in 79 countries to demonstrate why global marketers should finetune such experiences in accordance with between-country cultural differences. The results show that different cultural factors can enhance or hamper engagement responses to cognitive and social online customer experiences and thus have actionable practical implications for prioritizing distinct localization strategies.
This thesis reviewed scholarship in the field of media and cultural studies with specific focus on audience studies. Literature review was focused on key themes such as data practices and experiences, data privacy, advertising surveillance and issues around informed consent by users of digital platforms. The review identified a gap in research about user understanding of data exchange with digital platforms and experiences with personalised advertising in India. A mix of qualitative research methods were used to evaluate contemporary experiences of participants engaging with their personal data. This research explored the process of converting digital data trail into customised advertising for smartphone owners and their incentives to participate in this exchange of data for access to app-based services. Industry Experts working in media agencies and marketing function of advertisers were interviewed to understand various sources of digital data that are used for creating affinity audiences. Owners of smartphones in Mumbai and Delhi reflected on their awareness of automated data collection, trust threshold for sharing sensitive data and understanding of advertising technology. Participants discussed the affirmative nature of digital platforms and described services that were critical to their day-to-day functioning. Participants had a gradation of trust and all the apps needed to clear thresholds to be considered reliable. Most of the popular digital platforms were highly trusted on data safety. Digital platforms that scored high on trust quotient were also believed to obfuscate relevant information about the usage and storage of their data in lengthy terms and conditions. Four main reasons stated for not investing time to understand the use of software as service contracts were, There Is No Control (TINC), Fear of Missing (FOMO) on popular apps, there is no time (TINT) to read the lengthy terms and conditions and there is no option (TINO) as there was no alternative to the apps. Participants outlined multiple instances where they saw advertising about topics that they had recently discussed in the vicinity of their smartphones. This led to a widespread belief that smartphone apps such as Facebook and Instagram were listening to their conversations. Industry participants were able to describe an extensive array of practices that were used to identify the right target audience for their campaign, using algorithms running on large databases built on behavioural and transactional data. These processes created highly accurate predictive abilities that enabled highly accurate profiling of Internet users. Some Everyday Users were aware of these processes and the analysis identified them as algorithm-aware. Others were not equipped or motivated to discover information to understand advertising technology. In the absence of this information, participants used heuristics to understand the ability of digital platforms to deliver advertising that is so relevant to their current personal situation. This common-sense explanation for personalised advertising (Ads are listening to me) is named Folk Theory of Customised Advertising. The research highlights the enabling nature of digital technologies in India and outlines a requirement for an easy-to-use toolkit for everyday smartphone users to become algorithm-aware and privacy conscious.
A large-scale comparison of experimental advertising effects and those obtained using two state-of-the-art methods.
This research systemically reviews the directions in existing research in the digital marketing domain and unveils the irresponsibility in the digital advertising domain. The inefficiencies inherited from traditional advertising are enhanced or magnified by digital channels. This research reviews previous studies on advertising efficiency and states the enhanced challenges in the digital era: agency problem, advertising effect measurement, and the black box by programmatic advertising. Further, this research proposes the data as one potential direction for future study in the digital advertising domain.KeywordsIrresponsibilityDigital advertisingAdvertising efficiencyAgency problem
Online Display Advertising's importance as a marketing channel is partially due to its ability to attribute conversions to campaigns. Current industry practice to measure ad effectiveness is to run randomized experiments using placebo ads, assuming external validity for future exposures. We identify two different effects: a strategic effect of the campaign presence in marketplaces, and a selection effect due to user targeting, which are confounded in current practices. We propose two novel randomized designs to: 1) estimate the overall campaign attribution without placebo ads, 2) disaggregate the campaign presence and the ad effects. Using the Potential Outcomes Causal Model, we address the selection effect by estimating the probability of selecting influenceable users. We show the ex-ante value of continuing evaluation to enhance the user selection for ad exposure mid-flight. We analyze two performance-based (CPA) and one Cost-Per-Impression (CPM) campaigns with 20M+ users each. We estimate a negative CPM campaign presence effect due to cross product spillovers. Experimental evidence suggests that CPA campaigns incentivize the selection of converting users regardless of the ad, up to 96% more than CPM campaigns, thus challenging the standard practice of targeting most likely converting users.
Code:
https://github.com/joelbz/DispAdvAttr-in-Mrkt-ExpDgn-Est
External link:
https://users.soe.ucsc.edu/~jbarajas/publications/paper_MarketingScience.pdf
Firms track consumers’ shopping behaviors in their online stores to provide individually personalized banners through a method called retargeting. We use data from two large-scale field experiments and two lab experiments to show that, although personalization can substantially enhance banner effectiveness, its impact hinges on its interplay with timing and placement factors. First, personalization increases click-through especially at an early information state of the purchase decision process. Here, banners with a high degree of content personalization (DCP) are most effective when a consumer has just visited the advertiser’s online store, but quickly lose effectiveness as time passes since that last visit. We call this phenomenon overpersonalization. Medium DCP banners, on the other hand, are initially less effective, but more persistent, so that they outperform high DCP banners over time. Second, personalization increases click-through irrespective of whether banners appear on motive congruent or incongruent display websites. In terms of view-through, however, personalization increases ad effectiveness only on motive congruent websites, but decreases it on incongruent websites. We demonstrate in the lab how perceptions of ad informativeness and intrusiveness drive these results depending on consumers’ experiential or goal-directed Web browsing modes.
An analysis is performed on the results of 241 real world TV advertising tests conducted by Information Resources, Inc. between 1989 and 2003 to partially update the findings of Lodish et al. [Journal of Marketing Research 32, 2 (1995): 125-39]. Two types of market test results, BehaviorScan and Matched-Market, are analyzed. Overall, the improvement of TV advertising sales effectiveness because of media weight increase is significantly larger than zero for established products, which is different from Lodish et al.'s finding. A further analysis indicates that such significance is mainly driven by More recent tests. A comparison between the new results and Lodish et al. reveals a significant increase in the average advertising effectiveness for tests completed after 1995. The new data still suggest (as did the original data) that it is of great managerial interest to identify advertising effectiveness before launching advertising campaigns.
Social advertising uses information about consumers' peers, including peer affiliations with a brand, product, organization, etc., to target ads and contextualize their display. This approach can increase ad efficacy for two main reasons: peers' affiliations reflect unobserved consumer characteristics, which are correlated along the social network; and the inclusion of social cues (i.e., peers' association with a brand) alongside ads affect responses via social influence processes. For these reasons, responses may be increased when multiple social signals are presented with ads, and when ads are affiliated with peers who are strong, rather than weak, ties.
We conduct two very large field experiments that identify the effect of social cues on consumer responses to ads, measured in terms of ad clicks and the formation of connections with the advertised entity. In the first experiment, we randomize the number of social cues present in word-of-mouth advertising, and measure how responses increase as a function of the number of cues. The second experiment examines the effect of augmenting traditional ad units with a minimal social cue (i.e., displaying a peer's affiliation below an ad in light grey text). On average, this cue causes significant increases in ad performance. Using a measurement of tie strength based on the total amount of communication between subjects and their peers, we show that these influence effects are greatest for strong ties. Our work has implications for ad optimization, user interface design, and central questions in social science research.
Display advertising has traditionally been sold via guaranteed contracts – a guaranteed contract is a deal between a publisher and an advertiser to allocate a certain number of impressions over a certain period, for a pre-specified price per impression. However, as spot markets for display ads, such as the RightMedia Exchange, have grown in prominence, the selection of advertisements to show on a given page is increasingly being chosen based on price, using an auction. As the number of participants in the exchange grows, the price of an impressions becomes a signal of its value. This correlation between price and value means that a seller implementing the contract through bidding should offer the contract buyer a range of prices, and not just the cheapest impressions necessary to fulfill its demand.
Implementing a contract using a range of prices, is akin to creating a mutual fund of advertising impressions, and requires randomized bidding. We characterize what allocations can be implemented with randomized bidding, namely those where the desired share obtained at each price is a non-increasing function of price. In addition, we provide a full characterization of when a set of campaigns are compatible and how to implement them with randomized bidding strategies.
We investigate conditions sufficient for identification of average treatment effects using instrumental variables. First we show that the existence of valid instruments is not sufficient to identify any meaningful average treatment effect. We then establish that the combination of an instrument and a condition on the relation between the instrument and the participation status is sufficient for identification of a local average treatment effect for those who can be induced to change their participation status by changing the value of the instrument. Finally we derive the probability limit of the standard IV estimator under these conditions. It is seen to be a weighted average of local average treatment effects.
The authors analyze results of 389 BehaviorScan® matched household, consumer panel, split cable, real world T.V. advertising weight, and copy tests. Additionally, study sponsors—packaged goods advertisers, T.V. networks, and advertising agencies—filled out questionnaires on 140 of the tests, which could test common beliefs about how T.V. advertising works, to evaluate strategic, media, and copy variables unavailable from the BehaviorScan® results. Although some of the variables did indeed identify T.V. advertising that positively affected sales, many of the variables did not differentiate among the sales effects of different advertising treatments. For example, increasing advertising budgets in relation to competitors does not increase sales in general. However, changing brand, copy, and media strategy in categories with many purchase occasions in which in-store merchandising is low increases the likelihood of T.V. advertising positively affecting sales. The authors’ data do not show a strong relationship between standard recall and persuasion copy test measures and sales effectiveness. The data also suggest different variable formulations for choice and market response models that include advertising.
In a large-scale field experiment, we demonstrate that advertising can serve as a signal that enhances consumers' evaluations of advertised goods. We implement the experiment on a mobile search platform that provides listings and reviews for an archetypal experience good, restaurants. In collaboration with the platform, we randomize more than 200,000 consumers into exposure or no exposure of ads for about 600+ local restaurants. In conditions in which consumers are exposed to advertising, we also randomly vary the disclosure to the consumer of whether a restaurant's listing is an ad. This enables us to isolate the effect on outcomes of a consumer knowing that a listing is sponsored--a pure signaling effect. We find that this disclosure alone increases calls to the restaurant by 77%, holding fixed all other attributes of the ad. This effect is higher when the consumer uses the platform away from his typical city of search, when the uncertainly about restaurant quality is larger, and for restaurants that have received fewer ratings in the past. Further, on the supply side, newer, higher rated and more popular restaurants advertise more on the platform. Taken together, we interpret these results as consistent with a signaling equilibrium in which ads serve as implicit signals that enhance the appeal of the advertised restaurants. Both consumers and firms seem to benefit from the signaling. Consumers shift choices systematically towards restaurants that are better rated (at baseline) in the disclosure condition compared to the no disclosure condition, and advertisers gain from the improved conversion induced by disclosure. Further, our results imply that search-platforms would gain from clear sponsorship disclosure, and thus holds implications for platform design.
Identifying the same internet user across devices or over time is often infeasible. This presents a problem for online experiments, as it precludes person-level randomization. Randomization must instead be done using imperfect proxies for people, like cookies, email addresses, or device identifiers. Users may be partially treated and partially untreated as some of their cookies are assigned to the test group and some to the control group, complicating statistical inference. We show that the estimated treatment effect in a cookie-level experiment converges to a weighted average of the marginal effects of treating more of a user's cookies. If the marginal effects of cookie treatment exposure are positive and constant, it underestimates the true person-level effect by a factor equal to the number of cookies per person. Using two separate datasets---cookie assignment data from Atlas and advertising exposure and purchase data from Facebook---we empirically quantify the differences between cookie and person-level advertising effectiveness experiments. The effects are substantial: cookie tests underestimate the true person-level effects by a factor of about three, and require two to three times the number of people to achieve the same power as a test with perfect treatment assignment.
Yahoo! Research partnered with a nationwide retailer to study the effects of online display advertising on both online and in-store purchases. We use a randomized field experiment on 3 million Yahoo! users who are also past customers of the retailer. We find statistically significant evidence that the retailer ads increase sales 3.6% relative to the control group. We show that control ads boost measurement precision by identifying and removing the half of in-campaign sales data that are unaffected by the ads. Less data give us 31% more precision in our estimates—equivalent to increasing our sample to 5.3 million users. By contrast, we only improve precision by 5% when we include additional covariate data to reduce the residual variance in our experimental regression. The covariate-adjustment strategy disappoints despite exceptional consumer-level data including demographics, ad exposure levels, and two years’ worth of past purchase history.
Data, as supplemental material, are available at http://dx.doi.org/10.1287/mksc.2016.0998 .
The author analyzes the impact of online ads on the advertiser's competitors, using data from randomized field experiments on a restaurant-search website. He finds that ads increase the chances of sales for nonadvertised restaurants significantly. The spillover benefits are concentrated on restaurants that serve the advertiser's cuisine and have a high rating on the restaurant-search website. The extent of spillovers also depends on the intensity of the advertising effort. The spillovers are largest when the intensity (frequency) of advertising is low. As the intensity increases, the spillovers disappear and the advertiser gains more sales. These patterns are consistent with the following mechanism: ads increase the chance of consumers buying the advertised product but also remind consumers of similar (nonadvertised) options. Higher ad intensity leads to a stronger direct effect favoring the advertiser and can offset the spillover caused by the broader reminder.
Most questions in social and biomedical sciences are causal in nature: what would happen to individuals, or to groups, if part of their environment were changed? In this groundbreaking text, two world-renowned experts present statistical methods for studying such questions. This book starts with the notion of potential outcomes, each corresponding to the outcome that would be realized if a subject were exposed to a particular treatment or regime. In this approach, causal effects are comparisons of such potential outcomes. The fundamental problem of causal inference is that we can only observe one of the potential outcomes for a particular subject. The authors discuss how randomized experiments allow us to assess causal effects and then turn to observational studies. They lay out the assumptions needed for causal inference and describe the leading analysis methods, including, matching, propensity-score methods, and instrumental variables. Many detailed applications are included, with special focus on practical aspects for the empirical researcher.
Twenty-five large field experiments with major U.S. retailers and brokerages, most reaching millions of customers and collectively
representing $2.8 million in digital advertising expenditure, reveal that measuring the returns to advertising is difficult.
The median confidence interval on return on investment is over 100 percentage points wide. Detailed sales data show that relative
to the per capita cost of the advertising, individual-level sales are very volatile; a coefficient of variation of 10 is common.
Hence, informative advertising experiments can easily require more than 10 million person-weeks, making experiments costly
and potentially infeasible for many firms. Despite these unfavorable economics, randomized control trials represent progress
by injecting new, unbiased information into the market. The inference challenges revealed in the field experiments also show
that selection bias, due to the targeted nature of advertising, is a crippling concern for widely employed observational methods.
JEL Codes: L10, M37, C93.
This study examines the effects of Internet display advertising using cookie-level data from a field experiment at a financial tools provider. The experiment randomized assignment of cookies to treatment (firm ads) and control conditions (charity ads), enabling the authors to handle different sources of selection bias, including targeting algorithms and browsing behavior. They analyze display ad effects for users at different stages of the company's purchase funnel (i.e., nonvisitor, visitor, authenticated user, and converted customer) and find that display advertising positively affects visitation to the firm's website for users in most stages of the purchase funnel, but not for those who previously visited the site without creating an account. Using a binary logit model, the authors calculate marginal effects and elasticities by funnel stage and analyze the potential value of reallocating display ad impressions across users at different stages. Expected visits increase almost 10% when display ad impressions are partially reallocated from nonvisitors and visitors to authenticated users. The authors also show that results from the controlled experiment data differ significantly from those computed using standard correlational approaches.
We find display advertising influences customer search for both the advertised brand and its competitors. We exploit a natural experiment that randomizes ad delivery on 500 million visits to the Yahoo! homepage and compare visitors’ subsequent activities on Yahoo! Search. In three advertisers’ campaigns, display ads increase searches for advertised brands by 30-45 % and for competitors’ brands by up to 23 %. Strikingly, the total number of incremental searches for competitors is 2-8 times the increase for advertisers’ brands. We discuss how these spillovers create strategic complementarities for search advertisers and reduce firms’ investments in advertising.
Mobile advertising is one of the fastest-growing advertising formats. In 2013, global spending on mobile advertising was approximately 62.8 billion by 2017. The most prevalent type of mobile advertising is mobile display advertising (MDA), which takes the form of banners on mobile web pages and in mobile applications. This article examines which product characteristics are likely to be associated with MDA campaigns that are effective in increasing consumers' (1) favorable attitudes toward products and (2) purchase intentions. Data from a large-scale test-control field experiment covering 54 U.S. MDA campaigns that ran between 2007 and 2010 and involved 39, 946 consumers show that MDA campaigns significantly increased consumers' favorable attitudes and purchase intentions only when the campaigns advertised products that were higher (vs. lower) involvement and utilitarian (vs. hedonic). The authors explain this finding using established theories of information processing and persuasion and suggest that when MDAs work effectively, they do so by triggering consumers to recall and process previously stored product information.
Internet advertising has been the fastest growing advertising channel in recent years, with paid search ads comprising the bulk of this revenue. We present results from a series of large-scale field experiments done at eBay that were designed to measure the causal effectiveness of paid search ads. Because search clicks and purchase intent are correlated, we show that returns from paid search are a fraction of non-experimental estimates. As an extreme case, we show that brand keyword ads have no measurable short-term benefits. For non-brand keywords, we find that new and infrequent users are positively influenced by ads but that more frequent users whose purchasing behavior is not influenced by ads account for most of the advertising expenses, resulting in average returns that are negative.
A randomized experiment with 1.6 million customers measures positive causal effects of online advertising for a major retailer. The advertising profitably increases purchases by 5%. 93% of the increase occurs in brick-and-mortar stores; 78% of the increase derives from consumers who never click the ads. Our large sample reaches the statistical frontier for measuring economically relevant effects. We improve econometric efficiency by supplementing our experimental variation with non-experimental variation caused by consumer browsing behavior. Our experiment provides a specification check for observational difference-in-differences and cross-sectional estimators; the latter exhibits a large negative bias three times the estimated experimental effect.
Online experiments are widely used to compare specific design alternatives,
but they can also be used to produce generalizable knowledge and inform
strategic decision making. Doing so often requires sophisticated experimental
designs, iterative refinement, and careful logging and analysis. Few tools
exist that support these needs. We thus introduce a language for online field
experiments called PlanOut. PlanOut separates experimental design from
application code, allowing the experimenter to concisely describe experimental
designs, whether common "A/B tests" and factorial designs, or more complex
designs involving conditional logic or multiple experimental units. These
latter designs are often useful for understanding causal mechanisms involved in
user behaviors. We demonstrate how experiments from the literature can be
implemented in PlanOut, and describe two large field experiments conducted on
Facebook with PlanOut. For common scenarios in which experiments are run
iteratively and in parallel, we introduce a namespaced management system that
encourages sound experimental practice.
The Lipid Research Clinics Coronary Primary Prevention Trial (LRC-CPPT) measured the effectiveness of the drug cholestyramine for lowering cholesterol levels. The patients in the study were measured for compliance (the proportion of the intended dose actually taken) and for cholesterol decrease. The compliance-response regression for the Treatment group shows a smooth increasing effect of the drug in cholesterol level with increasing compliance. However, a similar, though less dramatic, compliance-response regression is seen in the Control group. This article investigates the recovery of the true dose-response curve from the Treatment and Control compliance-response curves. A simple model is proposed, analyzed, and applied to the LRC-CPPT data. Under this model, part but not all of the true dose-response curve can be estimated.
"We use a controlled field experiment to investigate the dynamic effects of retail advertising. The experimental design overcomes limitations hindering previous investigations of this issue. Our study uncovers dynamic advertising effects that have not been considered in previous literature. We find that current advertising does affect future sales, but surprisingly, the effect is not always positive; for the firm's best customers, the long-run outcome may be negative. This finding reflects two competing effects: brand switching and intertemporal substitution. We also find evidence of cross-channel substitution, with the firm's best customers switching demand to the ordering channel that corresponds to the advertising. "("JEL "L2, L81, M3) Copyright (c) 2008 Western Economic Association International.
Where Ads Might Appear in the Display Network
- Sahni Navdeep
- Goldfarb Avi
Measuring the Effectiveness of Online Advertising,” technical report, IAB France and SRI
- Pricewaterhousecoopers
How We're Making Ad Measurement More Insightful,” Facebook for Business News
When Money Moves to Digital, Where Should It Go?” comScore white paper
- Jacobsenmeredith Hunteranne
Display Retargeting Buyer's Guide,” technical report
- Econsultancy
The Online Display Ad Effectiveness Funnel and Carryover: Lessons from 432 Field Experiments
- A Johnsongarrett
- A Lewisrandall
State of the Industry: A Close Look at Retargeting and the Programmatic Marketer,” technical report
- Adroll
Machine Learning and the Facebook Ads Auction,” presentation in Joint Session for EC
- Q Candelajoaquin
- Dominowskaewa Baileymichael
Cross Channel Effects of Search Engine Advertising on Brick and Mortar Retail Sales: Insights from Multiple Large Scale Field Experiments on Google
- Mcateerjohn Kalyanamkirthi
- Marekjonathan
- A Hodgesjames
Worn-Out or Just Getting Started? The Impact of Frequency in Online Display Advertising
- A Lewisrandall
Understanding the Impact of Twitter Ads Through Conversion Lift Reports
- Shrivastavaa
Star Digital: Assessing the Effectiveness of Display Advertising
- Yildizt Narayanans