5 Common AB Testing Pitfalls
You had the best intentions.
Your shiny, new AB test idea was vetted by the appropriate teams, approved by stakeholders and is now ready to push live to production. After jumping through all necessary hoops in your organization’s extensive (and often cumbersome) AB testing process, the finished product seems pretty failproof. “What could possibly go wrong?” you ask yourself.
Unfortunately, many marketing managers and AB test leads can easily overlook some very basic checkpoints or make seemingly small missteps in the testing and optimization process. It happens to all of us. And without continuously scrutinizing your process, these minor miscues can undermine the integrity of the test altogether, wasting both time and money.
Let’s take a look at some common AB testing pitfalls, and tips on how we can avoid them.
1. You attempt to answer too many questions with one test
You are the optimization manager of an online produce company, The Melon Camp. Jack, a product owner, wants to know if altering the login button copy on the top of the homepage will encourage more existing customers to authenticate. Diane, a UX manager, is interested in knowing how a different color scheme in the global navigation will impact user engagement. In an effort to push both initiatives forward you decide to combine these two ideas into one AB test.
After a few weeks, your analyst tells you that the test experience drove much higher engagement, but isn’t sure what variables individually drove this improved performance. What initially seems like a terrific win results in the discouraging realization that a follow-up test is now needed to clarify the learnings. Rather than attempting to lump ideas together to save time, tests should carefully isolate the variables being measured to yield more meaningful results. If traffic volume is substantial enough, the two tests could run simultaneously in separate swimlanes, both gathering data at the same time but avoiding overlap that will ultimately pollute the data.
An example of a method used to separate similar traffic groups into swimlanes is to leverage the Visitor Profile attribute “user.userNumber” or “randomNumber” with a profile script, then assign values 1-49 to one test and 50-100 to the other.
2. Your AB test is targeted too granularly
It’s been roughly two weeks since the inception of your fancy, new homepage redesign test. Everyone on the team is psyched to find out how it’s performing and whether the test experience is a winner. With much anticipation, you scramble to your analytics platform and refresh the results only to find the significance level far below the desired threshold. “It’s a massive change!” you think to yourself… “We completely redesigned the Melon Camp header to include imagery of cantaloupe and watermelons, and changed the entire color scheme,” how could such drastic changes not result in higher confidence, you wonder.
Your analyst then reminds you that your test is only targeting residents in the Florida panhandle who have recently abandoned their carts. This initially seemed like a prudent method to glean specific learnings on a potentially profitable audience, but as a result, you are left with a test that’s collecting very low traffic volume. “Over-targeting” can result in an AB test that will never reach the required significance, thus wasting your team’s time and resources. Remember to do your due diligence measuring anticipated traffic volumes and expected length to completion to ensure the targeting is appropriate. A good tool to use is the Adobe Sample Size Calculator found here. If your goal is to learn more about very granular audiences within a test, wait until its completion, then slice and dice those audiences in Analytics to get a better understanding of how these groups performed on an individual level.
3. Once a winner, always a winner?
John, Melon Camp’s VP of Digital, once recommended an AB test that involved targeting “small towns” with unique messaging and calls-to-action in the homepage hero banner. After serving alternative copy and CTA verbiage to these rural customers, hero engagement and cart additions quickly skyrocketed. This was a big win, and the rural vs urban targeting strategy soon became a best practice within the MC digital marketing group.
Over time, several competing hypotheses were suggested to test the homepage hero copy further, but were consistently met with reluctance and trepidation. It’s not uncommon for successful test strategies to rapidly morph into corporate protocol. Just because a test experience won in the past does not mean it will win again in the future. Always stay flexible, continually challenging past beliefs and the status quo. Continuously evolving and iterating on past learnings is essential to the ongoing success of an optimization program.
4. You’re fixated on running only the “perfect” tests
The only “bad” AB test is the one you never actually get out the door. A common misstep in the AB testing process is simply not pushing enough ideas and variants forward, falling into the misconception that the idea or overarching strategy driving the idea must be perfect. Quite often the simplest, least sexy test ideas are the ones driving substantial business results.
This pitfall also includes the oversight that occurs when simple website updates are made without AB testing first. Any time a visual or navigational modification is made, there is an inherent opportunity to learn more about your visitors. Throttle the change to only a portion of traffic to ensure this new digital experience is actually a positive one in terms of your business goals. Split testing all significant site updates and feature rollouts is an easy way to exponentially ramp up your test volume, thus increasing your chances at identifying winning experiences faster while demonstrating the overall value of the AB test program to the organization.
5. You overlooked mobile
Recently, in what was viewed as an inevitability, Google announced that mobile traffic finally surpassed desktop traffic on its search engine. The consumer research group, Statista, estimates that the average time Americans spend on mobile devices per day is roughly three and a half hours. Depending on industry and target market these data points can be even more eye-popping, as millennials are estimated to stare at their devices for upwards of four hours daily!
Obviously these trends aren’t slowing, and marketers need to be more savvy than ever in tailoring their digital experiences for the handheld screen.
Either due to platform familiarity, or perhaps the default native functionality of many optimization tools, digital marketers too often overlook the mobile user when designing AB tests. This oversight can be an issue in both the planning and configuration phases of AB testing, as well in the performance reporting and analysis.
Adobe Target offers some simple tools and capabilities that can make this multiplatform optimization practice a bit more seamless, including the mobile viewport toggling within the Visual Experience Composer (VEC) and various device targeting capabilities within the audience builder, such as device model and screen size.
Lastly, never forget to segment your reporting by mobile traffic at the conclusion of your AB tests. You may learn something that surprises you.
