Split Testing – Make No Assumptions

Split testing (A/B and Multivariate) is a staple of good Internet marketing and it is an essential tool to improving marketing performance.

However, like everything related to online marketing, it’s not always as straight forward as determining a metric, serving up different creative, letting the test run and voila, a winner is declared, or sometimes not, after a period of time.

For relatively simple tests, such as whether the green button gets more clicks than the red button or whether text ad A gets more clicks than text ad B, it is possible to test quickly and move to the next test. Once you begin testing very different creative you run in to the laws of unintended consequences.

Testing two or more very different creative is similar to the introduction of new policy legislation, it changes user behavior, sometimes in unintentional ways. Similar to the policymaker, the marketer is hoping that the new creative (policy) has the desired impact (improved sales, better brand recall etc) but it is not always obvious that the outcome is being achieved. Very different creative with different calls to action (CTAs) change how your customer interacts and relates to your brand and this different “experience” with your brand can have a longer term impact beyond what the initial metrics or observations imply.

This realization came to me while I was conducting a landing page test recently.

The objective of our test was to improve the bounce rate while at a minimum not reducing our leads and sales. While we knew that the bounce rate would be determined relatively quickly, we were willing to let the test run to understand the impact upon leads and sales. Our logic was that a lower bounce rate and by correlation, a greater engagement with the brand would, in time, result in more leads and sales.

The Original vs The “New” Test Creative

The Results

There was a 30% decrease in the bounce rate. Unquestionably the new page was surpassing our expectations.

For leads, our second metric, we also saw an improvement of 8%.

It appeared that we had a winner on our hands until we looked at the sales numbers which were significantly better for the original page. Because we have a long sales cycle we divided our data sets to just look at new visitors (those who purchased in the same session) and to exclude returning visitors who are more inclined to purchase because of repeated exposure to the brand.

What we found was that the “new” visitors from the original landing page were also converting at twice the rate of the “new” visitors from the new landing page.

We had a dilemma. Two out of our three metrics were performing but the most important metric, sales, was not supporting the new landing page.

My marketing hat told me that the sales numbers were an anomaly and that this would be corrected in time. If our customers were engaging more with the brand (clicking to the next page and taking an action) and if they were more inclined to leave an email address because of the new landing page, then in time it would lead to more sales.

But I could not ignore the sales numbers. Because of this we began to look at how the different landing page cohorts were using the site. The customers to the new landing page were bouncing less and taking different paths through the site, but was that because we had limited the navigation to one principal call-to-action (CTA) where as the original landing page had numerous CTAs. We also theorized that because there was a video on the original landing page that our customers were getting all the information they needed from this video and for a number of people this was a more compelling reason to purchase than our new simplified and more emotionally driven landing page.

With contradictory data we decided to let the test run and sure enough in time the sales began to slowly switch in favor of the new landing page.

Finally we could declare a winner and we could switch to a new landing page.

Our initial data supported classic marketing theory – more engagement = more sales. We discovered on further analysis that the original landing page was reporting more sales because many of the sales reported to “new visitors” were in fact from returning users. We were able to determine this by the fact that many of those “new users” came from internal emails and that the problem was with the cookies or lack of.

Time and further analysis of the data proved that the initial success with engagement was translating in to better sales.

We can declare a winner but make no assumptions

What if the sales numbers had not turned in favor of the new landing page? The new page did change behavior and you can never make assumptions from the initial data. For tests with markedly different creative it is better to have a few data points and if in doubt don’t jump to assumptions and keep digging.