Incrementality Measurement: Ghost Ads
Our Data Science & Analytics team has been championing the benefits of using incrementality measurement as the ideal tool for determining marketing campaign effectiveness for a while now. We have previously written about the subject on a few occasions. You can read these previous blog posts for an introduction to the topic or a quick refresher before traveling further down the rabbit hole:
The basic concept of incrementality is simple: compared to a baseline of customer activity that would occur naturally, what is the actual effect of your marketing spend? When measuring incrementality, marketing professionals need to evaluate the performance of their advertising campaigns on targeted user populations, against control groups that did not receive the advertisement.
There has been new research done in the field of incrementality measurement, most notably regarding the use of “Predicted Ghost Ads” and the actualized benefits of this methodology compared to the two most common incrementality measurement experiments:
Public Service Announcements (PSAs) and “Intent-To-Treat” (ITTs) assignment designs.
The pros and cons of these incrementality measurement designs are examined at length in a paper titled, “Ghost Ads: Improving the Economics of Measuring Ad Effectiveness”. These experiments are basically A/B tests meant to observe a difference in behavior between groups, but the major difference between the two methodologies is regarding how their respective treatment groups and control groups are constructed. The researchers also conducted their own analyses of the aforementioned methodologies and compared them to their “Predicted Ghost Ads” methodology for measuring incrementality. The abstract from their research paper summarizes the benefits quite nicely, stating:
“… relative to public service announcement and intent-to-treat A/B tests, ghost ads can reduce the cost of experimentation, improve measurement precision, deliver the relevant strategic baseline, and work with modern ad platforms that optimize ad delivery in real time… They show novel evidence that retargeting can work: the ads lifted website visits by 17.2% and purchases by 10.5%. Compared with intent-to-treat and public service announcement experiments, advertisers can measure ad lift just as precisely while spending at least an order of magnitude less.”
At this point, you are probably wondering to yourself: how are they able to achieve these remarkable results that they are claiming? Again, the answer is found in how the treatment and control groups are constructed and deployed during the testing phase.
Figure 1: Johnson, G. A., Lewis, R. A., & Nubbemeyer, E. I. (2017). Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness. Journal of Marketing Research, 54(6), 867–884. https://doi.org/10.1509/jmr.15.0297
The Ghost Ads methodology works because unlike PSAs or ITTs it doesn’t serve ads to the control group at all; it only collects log of the auction bids that would have been served. A major component of this method is designating a subset of bids as “would-be-impressions” from the log of control group bids. These log entries function as counterfactual observations for the control group but also don’t incur any costs and are only measured at the exposure level, which allows the treatment and control group sizes to be kept consistent. This is achievable by using a simulated auction in combinations with a recall-optimized machine-learning model that is trained using the bid and impression data from the treatment group. By “recalling” most impressions in this way, it assists the test in capturing more conversions within the relevant population of the treatment and control groups while also helping to achieve significance.
This is a new frontier of incrementality measurement, and it is already gaining sweeping popularity in the marketing data science community due to the increased measurement precision with the added benefit of a lower testing cost. If you are interested in learning more about the topic of incrementality measurement or would like to schedule a free Incrementality Workshop with Softcrylic, please feel free contact us anytime by sending an email to: info@softcrylic.com
