Our 7 Steps CRO Framework & Why Ease Over Choice Always Wins in CRO?
Hey, guys! Up until this moment I (Darius) have been creating all the content for the AdKings Memos, but this memo will be special. It signifies the start of our team sharing their content and ideas. This Memo has been created by our team member, our Head of CRO - Sanmeet Walia.
He is a crazy guy with years of experience in working with the world's biggest businesses like Vodafone, Sainsbury's or Citibank and running CRO teams.
If I would have to describe him, the best description would be a lifelong learner and self-improvement book fanatic that can't spend a day without reading.
Anyway, giving a word over to him!
This case study will cover our 7-steps CRO framework and explains how one A/B test for our eCommerce partner right now is generating an additional $108K revenue.
It touches the classic paradox of choice - more is good, but not always.
This experiment was done for one of our partners who offers a wide variety of Athleisure wear. There are quite a few niche categories on their site like Jumpsuits, Anti-cellulite Leggings etc.
On this site, the average visitor browses 10-12 different products before converting, and perhaps this is the reason that our partner had decided to prioritize an exhibition of variety over easy access to product categories in the menu.
We at Adkings are fanatics over data. Be it ads or CRO; we rely heavily on rapid experimentations and exhaustive deep-dive data analysis to continually discover better variants of our ad campaigns and user experience.
For CRO we follow the following steps:
1. Implement a solid data foundation of precise tracking for quantitative analysis
2. Implement a layer of tools for qualitative analysis
3. Analyze the customer journeys
4. Form hypothesis
5. Design experiments to test the hypothesis
6. Analyze the experiment results
7. Calculate the impact
We followed the same steps for this case as well.
Step 1 - A solid data foundation for quantitative analysis
We are utilizing the combination of Google Analytics and Google Tag Manager for this. We have leveraged Google Analytics' enhanced eCommerce tracking to get granular data for product performance. On top of this, we have added numerous custom data points like scroll depths tracking, interactions with key CTAs etc.
Step 2 - A Layer of qualitative analysis
We are utilizing tools like HotJar for this. Qualitative analysis involves extracting insights from heatmaps, session recordings etc.
Step 3 - Customer journey analysis
Once we have the data ready, it's time to extract insights. We essentially aim to discover the frictions in the user journey and set benchmarks around the various macro and micro-conversion. All the experiments that we do after this phase will have the goal to perform better than the set benchmarks.
For this case study particularly, we discovered that plenty of users were clicking on the burger menu on the mobile device, but they were not engaging much after that. We created a segment in Analytics to compare the behavior of the users interacting with a menu on mobile vs. desktop, and we found that users interacting with a menu on the desktop were converting better than users interacting with a menu on mobile.
Step 4 - Form hypothesis
Based on the analysis we started considering that the sequence of items in the menu is suboptimal for the mobile experience, the current layout of the menu had laid a lot of focus on plenty of brand-specific collections rather than providing easy access to product categories.
The website receives a decent share of new visitors because of the Facebook ads we are running, and we were also concerned that new visitors might not be aware of the exclusive branded collections.
We formed this hypothesis that prioritizing product category access over collections will make it easier for users to navigate the site and would result in a lift in conversion rates.
Step 5 - Designing the experiment
We created a simple A/B test, where 50% of the visitors were served the existing version, and the other 50% were shown the modified version of the menu.
Step 6 - Analyzing the results
Since the menu is something that is universal on all the pages of the site, we considered transactions as the primary success metric for our experiment. We considered revenue and checkouts as our secondary metrics.
We ran the experiment for full 2 weeks and served nearly 20k sessions to both the variants of this experiment - the control and our variant.
For analysis, we not only just rely on the vanilla results by Google Optimize to declare our experiment a success or failure, but we have also created a Data Studio dashboard that segments the performance of the experiment by various dimensions like device category, region, marketing channel, browser type, visitor type(new/repeat) and few others.
But in this case, Google Optimize and all the other reports in Data Studio were calling out a clear winner with 96% probability of beating the existing variant.
But, the aha moment!
The visitors were not only converting better with the new variant but were also converting at a higher AOV!
It was simple - since users could discover more products easily, they were adding more to their cart as well.
Step 7 - Calculate the impact
The result - We got a variant that not only has a 6+% higher conversion rate but is converting at 5+% more AOV!
5%, 6% seem small numbers. But upon projecting the benefits over the entire year, this translates to incremental revenues of $108K annually for our partner!
By doing CRO, our partner is getting $108K of free revenue without extra spends in paid campaigns. Hence we heavily stress on CRO and constantly experimenting. We don't expect every experiment to win, but even if we are able to discover a few winning variants over time, it creates a substantial long term impact.
Thank you for reading!
- Sanmeet Walia, Head of CRO at AdKings Agency