Building a Successful Optimization Practice: PROCESS
Developing a successful optimization practice within your organization doesn’t happen overnight. In this series we will examine the various components needed to design, build and maintain a successful personalization and optimization practice. Phase 2 of this series covers the processes necessary to ensure a profitable program.
If you’ve already read the first installment of this series, then you’re familiar with our fictional friends at Pineapple Technologies. To recap, PineTech’s leadership team is eager to improve online sales for the new PinePhone X, and Cooper – the Optimization Lead – is solely responsible for developing and executing the digital strategy.
Cooper works closely with Kate, the Optimization Strategist. Kate is a seasoned Digital Marketing Manager at PineTech and is very familiar with omnichannel promotional tactics.
Ava – the designated Business Analyst on the optimization practice – is tasked with supporting the roadmap management and test planning, as well as being the point person for delivering valuable reporting analysis and insights from completed A/B tests.
For all coding expertise, the team leans heavily on Rose, a front-end developer who assumes the role of Technical Lead.
Lucy, the Creative Lead, is the creative mastermind behind the asset and experience design, while Riley holds down the testing side of things as QA Analyst.
Despite the vast and diverse expertise on this mythical optimization practice, the adherence to an efficient iterative process is vital to sustained success. Below I’ll outline the best practices of an iterative optimization process and the key stakeholders involved along the way.
Fundamentally cyclical in nature, there really isn’t a definitive starting or ending point to the optimization process, so for this exercise let’s begin with developing the test strategy.
Develop Strategy
The testing strategy may originate as a directive from the leadership team, or it can be the iterative byproduct of a previous A/B test. Regardless of source, testing strategies need to be founded on business objectives with clearly defined goals in mind.
At PineTech, for example, Kate was told by her director that it was important to test different creative for the PinePhone X launch to gather insights that can be leveraged for future product releases. Leadership was most interested in what features of the PinePhone drove the strongest onsite engagement. With this in mind, Kate huddled with the team and began building out testing plans that involved A/B testing the PinePhone creative on the homepage.
Brainstorm & Roadmap
Kate, Cooper and the optimization team began brainstorming test ideas. As ideas were discussed, they were added to the testing roadmap and assigned estimated business impact and technical effort scores. While not an exact science, the practice of scoring test ideas is an essential step in the prioritization process.
At the top of the list for the PineTech team was A/B testing hero banners with various features highlighted in both copy and imagery. With this test concept in mind, the team began building initial test plans.
Create Test Plan
The test plan itself documents all details necessary to build and execute an A/B test. It is essentially a mini requirements document outlining the hypothesis and expected outcome, the test location, targeting specifics and all assets required to launch the test. A project management system like Jira is a popular tool for compiling this information in an orderly fashion.
Cooper pulled these details together in a Test Summary document similar to the example below. It is also prudent to provide an estimated test duration during the test planning stage, as this assists in setting business expectations as well as coordinating multiple test priorities. Test duration estimates can be calculated by inputting expected volume and lift into a test calculator similar to this one.
Creative Development
With the test plan now in hand, Lucy – the optimization practice’s Creative Lead – can now begin asset development. Once these sexy new hero banners are designed and built, either as Experience Fragments in Adobe Experience Manager or with Rose’s assistance as HTML offers in Target, Lucy’s job is complete. These beauties can now be handed off to the QA team for final review.
Build & QA Campaign
Cooper receives a notification in Jira that all necessary assets are built and approved for use in production. First, he fist-pumps in sheer excitement, then proceeds to set up the A/B test in Adobe Target.
Once the test setup is complete, including the appropriate audience segmentation (if applicable), Cooper then tags the QA Lead – Riley – for production validation. Typically, the Optimization Lead will also share preview links with all other stakeholders as well at this stage. Once Riley signs off on the performance of the assets in production, and test stakeholders approve of the control and test experiences, Cooper is ready to launch the A/B test.
Launch Campaign
Finally! After rigorous prioritization, test planning and asset development, the test is now fully built and approved, ready for activation. Cooper ensures there are no potential activity collisions in the Target UI one last time, then launches the test to production. Communications to the rest of the PineTech stakeholders is conducted over email as well as updating the project status in Jira.
Analyze Results
The test has been live for two days when the emails begin to pour into Cooper’s Outlook inbox.
“How’s it looking? Is the new banner winning?”
“When can we call a winner?”
“What is the lift of the new hero?”
Sound familiar? Inevitably there will be strong interest in the outcomes of your A/B tests, especially when located in highly visible real estate like the homepage.
A best practice for streamlining the reporting schedule – while also maintaining your sanity – is to build templated performance dashboards for each A/B test and share these reports on a routine basis. Within Adobe Analytics, a Workspace report can quickly be shared by copying the report link itself, or by scheduling an email delivery cadence.
Back at PineTech headquarters, Ava – the practice’s Business Analyst – assists in setting up the Workspace for the homepage hero A/B test, including click-through-rate performance as well as PinePhone X onsite conversion. Cooper informs Ava which stakeholders to include in the email distribution list.
Promote Winner
Once the test reaches the preferred confidence level (ideally >95%), it is appropriate to conclude the test and ensure the winning experience is documented for iteration planning. Assuming the test surfaces a clear winner, this experience is ideally hardcoded to the website, becoming the “new” control experience.
After determining the PinePhone banners with additional imagery outperformed the control heroes, Ava and Cooper communicated these learnings to all stakeholders and concluded the test. Cooper then funneled these learnings back into the strategy and planning sessions he has with Kate and the rest of the optimization practice. The success of these banners on the homepage also influences the creative strategy leveraged on product pages and 3rd party ads, as well as in email campaigns.
TL;DR
Great teams can only go so far without strong processes. Each member of the optimization practice has a clearly defined role that directly contributes to the ongoing success of the iterative testing cycle.
Be sure to check back for the next installment of this series, where we’ll examine best practices crucial to long-term growth and sustainability of your optimization practice.
