Memo #14

The Secret to Actually A/B Testing Plugins in Shopify.


This memo will present you with a solution to a huge problem a lot of Shopify owners have.

A legitimate way to A/B test plugins on Shopify.

You see, generally, with Shopify, you can't really A/B test any plugins. Some of them do have that feature built-in, but the majority doesn't. Also, you're never really sure if they show real data...

This leaves you with a very interesting debate.

Does the plugin I've installed actually help?

Yes... I see that the upsell app is reporting new sales through it, or my AOV has increased, but what if your conversion rate dropped due to complicating the customer journey, and in the end, I'm actually losing money?

Tricky right?

Which tools are usually used?

There are many tools to perform A/B tests or other experiments, like Google Optimize, Optimizely, Adobe Target, Monetate etc.

Most of the tools come with a visual editor where one can make the changes to the page or set of pages by updating the HTML, CSS or JavaScript.

It is fairly simple to use these tools for testing cosmetic changes like CTA colors, rearrangement of content etc.

That is what usually is available to the majority of people.

Now, if you have a JavaScript programmer in the team, you can expand this functionality a great deal and even introduce new features like making the add to cart button sticky on the mobile experience or testing 2 completely different menu structures with full functionality. (This is why you need a full team for it!)

But, plugins?

Well, it took us a few weeks to figure it out. It took us some exploring, testing some ideas and crafting at least half a dozen different custom JavaScript codes.

But first, let me take you over the scope of the solution:

This solution will work for plugins that make additions to the existing page but not necessarily update the features of the page.

For example, we can test a plugin that "introduces" a sticky add to cart but does not alter the existing add to cart CTA on the page or we can test a plugin that introduces a timer on the page, but it should not apply a discount automatically.

To test the plugin, we do something different than the other experiments.

Since after enabling the plugin, your default/original experience becomes the one with a plugin, the variant will actually be the one without the plugin.

Hence, to assess the impact of a plugin, we will consider the original as the version with features to be tested and variant as the default.

To create the variant, you need to make it exactly the same as original but, write a custom JS script to hide the plugin.

Script for each plugin and store will be unique; this is why we're not sharing it.

A decent JavaScript programmer should be able to do it.

The trick is in implementing the test in A/B testing software.

You implement this script with any of the software mentioned above that has JS capability. You drop in this custom JS code it on the variant page via JS script insert and you have to make sure it fires on priority.

This way the plugin on the variant will not be shown to 50% of the people sent to the variant by the A/B testing software.


Now you will have a real A/B tests data and whatever that new fancy plugin is actually helping or killing your performance!

It's been super effective for us and helped to greatly elevate results we can bring to our clients with CRO efforts!

Thank you for reading!

- Darius Kunca, Co-founder at AdKings Agency

Ready to Take Your eCommerce Business to the Next Level?

Click the button below.

AdKings - White Banner Logo - LG - 1.05