David Holtz’s research while affiliated with University of California System and other places

What is this page?

This page lists works of an author who doesn't have a ResearchGate profile or hasn't added the works to their profile yet. It is automatically generated from public (personal) data to further our legitimate goal of comprehensive and accurate scientific recordkeeping. If you are this author and want this page removed, please let us know.

Publications (23)

As Generative Models Improve, People Adapt Their Prompts
  • Preprint

July 2024

·

6 Reads

Eaman Jahani

·

Benjamin S. Manning

·

Joe Zhang

·

[...]

·

David Holtz

In an online experiment with N = 1891 participants, we collected and analyzed over 18,000 prompts to explore how the importance of prompting will change as the capabilities of generative AI models continue to improve. Each participant in our experiment was randomly and blindly assigned to use one of three text-to-image diffusion models: DALL-E 2, its more advanced successor DALL-E 3, or a version of DALL-E 3 with automatic prompt revision. Participants were then asked to write prompts to reproduce a target image as closely as possible in 10 consecutive tries. We find that task performance was higher for participants using DALL-E 3 than for those using DALL-E 2. This performance gap corresponds to a noticeable difference in the similarity of participants' images to their target images, and was caused in equal measure by: (1) the increased technical capabilities of DALL-E 3, and (2) endogenous changes in participants' prompting in response to these increased capabilities. More specifically, despite being blind to the model they were assigned, participants assigned to DALL-E 3 wrote longer prompts that were more semantically similar to each other and contained a greater number of descriptive words. Furthermore, while participants assigned to DALL-E 3 with prompt revision still outperformed those assigned to DALL-E 2, automatic prompt revision reduced the benefits of using DALL-E 3 by 58%. Taken together, our results suggest that as models continue to progress, people will continue to adapt their prompts to take advantage of new models' capabilities.

Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb

April 2024

·

44 Reads

·

9 Citations

Management Science

David Holtz

·

Felipe Lobel

·

·

[...]

·

Online marketplace designers frequently run randomized experiments to measure the impact of proposed product changes. However, given that marketplaces are inherently connected, total average treatment effect (TATE) estimates obtained through individual-level randomized experiments may be biased because of violations of the stable unit treatment value assumption, a phenomenon we refer to as “interference bias.” Cluster randomization (i.e., the practice of randomizing treatment assignment at the level of “clusters” of similar individuals) is an established experiment design technique for countering interference bias in social networks, but it is unclear ex ante if it will be effective in marketplace settings. In this paper, we use a meta-experiment or “experiment over experiments” conducted on Airbnb to both provide empirical evidence of interference bias in online marketplace settings and assess the viability of cluster randomization as a tool for reducing interference bias in marketplace TATE estimates. Results from our meta-experiment indicate that at least 20% of the TATE estimate produced by an individual-level randomized evaluation of the platform fee increase we study is attributable to interference bias and eliminated through the use of cluster randomization. We also find suggestive, nonstatistically significant evidence that interference bias in seller-side experiments is more severe in demand-constrained geographies and that the efficacy of cluster randomization at reducing interference bias increases with cluster quality. This paper was accepted by Chris Forman, information systems. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2020.01157 .

Time trend comparisons a–d, The average number of bridging ties per month (a,c) and the average unscheduled video/audio call hours per week (b,d) for different groups of employees, relative to the overall average in February. These plots establish the plausibility of the ‘parallel trends’ assumption that is required by our modified DiD model. The error bars show the 95% CIs and are in some places thinner than the symbols in the figure; s.e. values are clustered at the team level. a,b, The graphs show employees who, before COVID-19, worked from the office (blue; n = 50,268) and a matched sample of employees who worked remotely (orange; n = 10,914). c,d, The graphs show two subgroups of the blue lines in a and b—employees who, before COVID-19, had less than 10% of their collaborators working remotely (dashed; n = 36,008) and those who had more than 50% of their coworkers working remotely (dotted; n = 1,861). Both variables were normalized by subtracting and dividing by the average across the entire sample of that variable in February. Most employees transitioned to WFH during the week of 1 March 2020, although our analysis omits the month of March as a transition period.
Time trends for collaboration networks a–k, The monthly averages for the collaboration network variables for all employees relative to the February average. Each variable was normalized by subtracting and dividing by the average FV for that variable. The vertical bars show the 95% CIs, but are in most places not much taller than the data points; s.e. values are clustered at the team level. The variables are employees’ average number of network ties (a), distinct business groups in which they have a collaborator (b), cross-group ties (c), ties that bridge structural holes in the network (e), individual clustering coefficient (g), collaborators from the previous month that they did not collaborate with that month (i) and added collaborators they did not collaborate with the previous month (j), as well as the share of time spent with cross-group ties (d), bridging ties (f), weak ties (h) and added ties (k). n = 61,279 for each panel.
Effects of firm-wide remote work on collaboration networks The estimated causal effects of both an employee and that employee’s colleagues switching to remote work on the number of collaborators an employee has, the number of distinct groups the employee collaborates with, the number of cross-group ties an employee has, the share of time an employee spends collaborating with cross-group ties, the number of bridging ties an employee has, the share of time an employee spends collaborating with bridging ties, the individual clustering coefficient of an employee’s ego network, the share of time an employee spent collaborating with weak ties, the number of churned collaborators, the number of added collaborators and the share of time spent with added collaborators. The reported effects are (β + δ) from equation (1), normalized by dividing by the average level of that variable in February. The symbols depict point estimates and the lines show the 95% CIs. n = 61,182 for all variables. The full results are provided in Supplementary Tables 1 and 2.
Time trends for communication media a–f, The weekly averages for each variable, relative to the February average. Each variable was normalized by subtracting and dividing by the average FV for that variable. The vertical bars show the 95% CIs, but are in most places not much taller than the data points; s.e. values are clustered at the team level. The variables are the employees’ average number of unscheduled audio/video call hours (a), scheduled meeting hours (b), total hours in scheduled meetings and unscheduled calls (the sum of a and b) (c), IMs sent (d), emails sent (e), and hours between the first and last activity (sent email, scheduled meeting, or Microsoft Teams call or chat) in a day, summed across the workdays (f). The dips in all six metrics during the weeks of 16 February, 24 May and 14 June were due to four-day workweeks, in observance of Presidents’ Day, Juneteenth and Memorial Day, respectively. n = 61,279 for all variables.
The effects of firm-wide remote work on the use of communication media The estimated causal effects of both an employee and their colleagues switching to remote work on the employee’s hours spent in scheduled meetings, hours spent in unscheduled calls, the sum of meetings and call hours, IMs sent, emails sent and estimated workweek hours. The reported effects are (β + δ) from equation (1), normalized by dividing by the average level of that variable in February. The symbols depict point estimates and lines depict 95% CIs. n = 61,182 for all variables. The full results are provided in Supplementary Table 3.

+3

The effects of remote work on collaboration among information workers
  • Article
  • Publisher preview available

January 2022

·

1,566 Reads

·

511 Citations

Nature Human Behaviour

Longqi Yang

·

David Holtz

·

Sonia Jaffe

·

[...]

·

Jaime Teevan

The coronavirus disease 2019 (COVID-19) pandemic caused a rapid shift to full-time remote work for many information workers. Viewing this shift as a natural experiment in which some workers were already working remotely before the pandemic enables us to separate the effects of firm-wide remote work from other pandemic-related confounding factors. Here, we use rich data on the emails, calendars, instant messages, video/audio calls and workweek hours of 61,182 US Microsoft employees over the first six months of 2020 to estimate the causal effects of firm-wide remote work on collaboration and communication. Our results show that firm-wide remote work caused the collaboration network of workers to become more static and siloed, with fewer bridges between disparate parts. Furthermore, there was a decrease in synchronous communication and an increase in asynchronous communication. Together, these effects may make it harder for employees to acquire and share new information across the network.

View access options
Figure 1: Treatment Email
Figure 2: Distribution of Ratings for Focal Stay
Figure 3: Cumulative Effects of Treatment on Listing Outcomes
Figure 4: Distribution of Days to First Review
Figure 5: Effect on Review Ratings (Conditional on Review)

+10

More Reviews May Not Help: Evidence from Incentivized First Reviews on Airbnb

December 2021

·

65 Reads

Andrey Fradkin

·

David Holtz

Online reviews are typically written by volunteers and, as a consequence, information about seller quality may be under-provided in digital marketplaces. We study the extent of this under-provision in a large-scale randomized experiment conducted by Airbnb. In this experiment, buyers are offered a coupon to review listings that have no prior reviews. The treatment induces additional reviews and these reviews tend to be more negative than reviews in the control group, consistent with selection bias in reviewing. Reviews induced by the treatment result in a temporary increase in transactions but these transactions are for fewer nights, on average. The effects on transactions and nights per transaction cancel out so that there is no detectable effect on total nights sold and revenue. Measures of transaction quality in the treatment group fall, suggesting that incentivized reviews do not improve matching. We show how market conditions and the design of the reputation system can explain our findings.

Download

Citations (15)

... Experimental design, also known as randomization scheme, decides the allocation of treatments to units. A classic methodology to tackle network interference is applying cluster-level randomization [36,64,65,34], instead of randomizing at the unit level. Intuitively, cluster-level randomization introduces strong correlations among those units connected densely and creates an environment mimicking global treatment in the interior of the cluster. ...

Reference:

Just Ramp-up: Unleash the Potential of Regression-based Estimator for A/B Tests under Network Interference
Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb
  • Citing Article

  • April 2024

Management Science

David Holtz

·

Felipe Lobel

·

·

[...]

·

... While in some datasets, e.g., PubMed and ArXiv, the RMSE seems to be small, it is important to also consider the effect size, because even a small error in the estimation can result in suboptimal decision-making. In large-scale marketing experiments, small effects (e.g., < 1%) can have significant implications for decision-making (Blake and Coey 2014;Fradkin and Holtz 2023). A low RMSE (e.g., 0.01) indicates precise predictions, facilitating the detection and estimation of small effects. ...

Reference:

Cascade-Based Randomization for Inferring Causal Effects under Diffusion Interference
Do Incentives to Review Help the Market? Evidence from a Field Experiment on Airbnb
  • Citing Article

  • April 2023

Marketing Science

Andrey Fradkin

·

David Holtz

... Digital innovation can be defined as " [...] the use of digital technology during the process of innovating" (Nambisan et al., 2017, p. 223). The primary challenges of the distributed environment that impact digital innovation are (1) Communication and collaboration: Distance and asynchronous communication can hinder effective collaboration and information sharing, which is crucial for generating and refining ideas (Garro-Abarca et al., 2021;Yang et al., 2022); (2) Lack of trust and psychological safety: Building trust and rapport in a virtual setting can be challenging, hindering open communication and risktaking, which are essential for innovation (Hacker et al., 2019;Hao et al., 2022); (3) Knowledge sharing and learning: Limited opportunities for informal interaction and knowledge exchange can create silos and hinder the transfer of tacit knowledge, which is vital for creativity and problem-solving (Davidavičienė et al., 2020); and (4) Lack of co-creation opportunities: The absence of physical proximity can limit the ability of team members to co-create and iterate on ideas in real-time, hindering the development of innovative solutions Salazar Miranda and Claudel, 2021). ...

Reference:

The Role of the Metaverse in Facilitating Innovation Processes within Distributed Work Settings
The effects of remote work on collaboration among information workers
  • Citing Article

  • September 2022

Yearbook of Paediatric Endocrinology

Yang L

·

Holtz D

·

Jaffe S

·

[...]

·

Teevan J

... The conventional economic literature defines the maximum amount an individual would be willing to pay to acquire an amenity (e.g., the work flexibility, the work assistance) as their willingness to pay (WTP) for the amenity [32,40], which can be intuitively comprehended as the value of the amenity to them. Researchers in the human-computer interaction community have previously worked on quantifying the value of various amenities [20,24,44]. According to the existing literature, the prevailing best practice for quantifying individuals' willingness to pay or accept for the amenity is through discrete choice experiments [22,30,32]. ...

Reference:

The Value, Benefits, and Concerns of Generative AI-Powered Assistance in Writing
How Much Do Platform Workers Value Reviews? An Experimental Method
  • Citing Conference Paper

  • April 2022

David M Holtz

·

·

Siddharth Suri

... Finally, it is important to consider that individuals who leave reviews are likely different from those who do not, and individuals who visit memorials are likely different from those who choose not to visit. For example, studies of AirBnB reviews have shown that those who do not leave reviews tend to have worse experiences than those who do (Fradkin et al., 2021). While this study controls for review ratings, which helps account for some of the bias in emotional expression, it is important to recognize that the dataset may not fully capture the range of experiences at memorial sites. ...

Reference:

Memorials and collective memory: A text analysis of online reviews
Reciprocity and Unveiling in Two-Sided Reputation Systems: Evidence from an Experiment on Airbnb

Marketing Science

Andrey Fradkin

·

Elena Grewal

·

David Holtz

... In the digital era that continues to grow, Information and Communication Technology has become an integral part of everyday life in the educational process. ICT has changed the way we communicate, access information and do work (Yang et al., 2021) . In the educational context, the use of ICT has great potential to improve the quality of learning and teacher productivity (Alam, 2022). ...

Reference:

Correlation of Digital Literacy and Teacher Productivity at SMP Negeri 1 Serang City
Author Correction: The effects of remote work on collaboration among information workers

Nature Human Behaviour

Longqi Yang

·

David Holtz

·

Sonia Jaffe

·

[...]

·

Jaime Teevan

... The pandemic catalyzed unprecedented shifts in human communication patterns, with a notable surge in videoconferencing [1][2][3]. This technology has become essential for interactions even after the pandemic, especially when in-person meetings are impractical. ...

Reference:

Too much to process? Exploring the relationships between communication and information overload and videoconference fatigue
The effects of remote work on collaboration among information workers

Nature Human Behaviour

Longqi Yang

·

David Holtz

·

Sonia Jaffe

·

[...]

·

Jaime Teevan

... at cross-boundary economic activities are not explored. The study considers only interaction within the specific administrative boundary. The challenge with this assumption is that economic interaction within an area might potentially impact neighbouring areas' interaction patterns due to geographical interdependency (Fotheringham & Rogerson, 1993;M. Zhao et al., 2021). The study's analysis considers only the job location and crime within the boundaries of main places within a metro and, therefore, disregards the effect of cross-boundary economic activity. Although distance decay will probably result in reduced activities in the periphery, people might still travel to jobs in neighbouring municipaliti ...

Reference:

Examining the impact of job location on violent crime: the study of South African metropolitan areas
Interdependent program evaluation: Geographic and social spillovers in COVID-19 closures and reopenings in the United States

Science Advances

Michael Zhao

·

David Holtz

·

... For example, the limited capacity of properties in peer-to-peer rental platforms has resulted in a relatively scant number of reviews compared with hotel booking platforms (Liang, Schuckert, Law, & Chen, 2020). Some studies have also indicated the existence of review or rating bias on peer-to-peer rental platforms, such as Airbnb (Holtz & Fradkin, 2020;Pera, Viglia, Grazzini, & Dalli, 2019). Zervas, Proserpio, and Byers (2021) compared the rating distribution between Airbnb properties and TripAdvisor hotels and found that the ratings in Airbnb are relatively much higher and have lower variations than those in TripAdvisor. ...

Reference:

How does constraining description affect guest booking decisions and satisfaction?
Tit for Tat? The Difficulty of Designing Two-Sided Reputation Systems

NIM Marketing Intelligence Review

David Holtz

·

Andrey Fradkin

... Diversity.Systemic Volume. Individual Holtz et al. [77] examine the impact of personalised recommendations on podcast consumption among approximately 900,000 Spotify premium users across seventeen countries. Users in the treatment group are exposed to personalised recommendations based on their historical listening behaviour, while those in the control group are exposed to the most popular podcasts. ...

Reference:

A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions
The Engagement-Diversity Connection: Evidence from a Field Experiment on Spotify
  • Citing Conference Paper

  • July 2020

David Holtz

·

·

·

[...]

·