Randal M. Henne’s research while affiliated with Microsoft and other places

What is this page?


This page lists works of an author who doesn't have a ResearchGate profile or hasn't added the works to their profile yet. It is automatically generated from public (personal) data to further our legitimate goal of comprehensive and accurate scientific recordkeeping. If you are this author and want this page removed, please let us know.

Publications (2)


Figure 2: Microsoft Help Ratings Widget. The original widget is shown above. When users click on Yes/No, the dialogue continues asking for free-text input (two-phase)
Figure 6: Amazon search for "24" with and without BBS
Figure 7: High-level flow for an A/B test
Controlled experiments on the web: Survey and practical guide
  • Article
  • Full-text available

February 2009

·

4,612 Reads

·

711 Citations

Data Mining and Knowledge Discovery

·

·

Dan Sommerfield

·

Randal M. Henne

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, Control/Treatment tests, MultiVariable Tests (MVT) and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person’s Opinion (HiPPO). We provide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for variance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. We evaluate randomization and hashing techniques, which we show are not as simple in practice as is often assumed. Controlled experiments typically generate large amounts of data, which can be analyzed using data mining techniques to gain deeper understanding of the factors influencing the outcome of interest, leading to new hypotheses and creating a virtuous cycle of improvements. Organizations that embrace controlled experiments with clear evaluation criteria can evolve their systems with automated optimizations and real-time analyses. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.

Download

Figure 2 
Practical guide to controlled experiments on the web: Listen to your customers not to the hippo

August 2007

·

2,153 Reads

·

370 Citations

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments (single-factor or factorial designs), A/B tests (and their generalizations), split tests, Control/Treatment tests, and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-on- investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person's Opinion (HiPPO). We provide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for variance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. We evaluate randomization and hashing techniques, which we show are not as simple in practice as is often assumed. Controlled experiments typically generate large amounts of data, which can be analyzed using data mining techniques to gain deeper understanding of the factors influencing the outcome of interest, leading to new hypotheses and creating a virtuous cycle of improvements. Organizations that embrace controlled experiments with clear evaluation criteria can evolve their systems with automated optimizations and real-time analyses. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.

Citations (2)


... It is not clear what delivering value to the client really means and this, as a consequence, impede to draw a data-driven prioritization framework. A common practice is that product backlog items are prioritized by management or expert opinions, sometimes causing the HiPPO (Highest Paid Person Opinion) effect (Münch et al., 2019b;Kohavi et al., 2007). The risk with this approach is that the wrong priorities are set because the criteria for decision-making are unclear (Münch et al., 2019b). ...

Reference:

Criteria definition for digital requirements using hesitant fuzzy linguistic terms sets: an application to the automotive industry
Practical guide to controlled experiments on the web: Listen to your customers not to the hippo

... Thousands of Microsoft employees use ExP every day to test ideas through A/B tests aka experiments. In total, there are approximately one hundred thousand A/B tests run per year now through ExP, which is a scale that we've achieved over nearly 20 years of growth [1], [2]. Conservative estimates in prior research [3] suggest that A/B tests confirm expectations only a third of the time. ...

Controlled experiments on the web: Survey and practical guide

Data Mining and Knowledge Discovery