Split Testing – Make No Assumptions
Split testing (A/B and Multivariate) is a staple of good Internet marketing and it is an essential tool to improving marketing performance.
However, like everything related to online marketing, it’s not always as straight forward as determining a metric, serving up different creative, letting the test run and voila, a winner is declared, or sometimes not, after a period of time.
For relatively simple tests, such as whether the green button gets more clicks than the red button or whether text ad A gets more clicks than text ad B, it is possible to test quickly and move to the next test. Once you begin testing very different creative you run in to the laws of unintended consequences.
Testing two or more very different creative is similar to the introduction of new policy legislation, it changes user behavior, sometimes in unintentional ways. Similar to the policymaker, the marketer is hoping that the new creative (policy) has the desired impact (improved sales, better brand recall etc) but it is not always obvious that the outcome is being achieved. Very different creative with different calls to action (CTAs) change how your customer interacts and relates to your brand and this different “experience” with your brand can have a longer term impact beyond what the initial metrics or observations imply.
This realization came to me while I was conducting a landing page test recently.
The objective of our test was to improve the bounce rate while at a minimum not reducing our leads and sales. While we knew that the bounce rate would be determined relatively quickly, we were willing to let the test run to understand the impact upon leads and sales. Our logic was that a lower bounce rate and by correlation, a greater engagement with the brand would, in time, result in more leads and sales.
The Original vs The “New” Test Creative
Original Page
Test Page
The Results
There was a 30% decrease in the bounce rate. Unquestionably the new page was surpassing our expectations.
For leads, our second metric, we also saw an improvement of 8%.
It appeared that we had a winner on our hands until we looked at the sales numbers which were significantly better for the original page. Because we have a long sales cycle we divided our data sets to just look at new visitors (those who purchased in the same session) and to exclude returning visitors who are more inclined to purchase because of repeated exposure to the brand.
What we found was that the “new” visitors from the original landing page were also converting at twice the rate of the “new” visitors from the new landing page.
We had a dilemma. Two out of our three metrics were performing but the most important metric, sales, was not supporting the new landing page.
My marketing hat told me that the sales numbers were an anomaly and that this would be corrected in time. If our customers were engaging more with the brand (clicking to the next page and taking an action) and if they were more inclined to leave an email address because of the new landing page, then in time it would lead to more sales.
But I could not ignore the sales numbers. Because of this we began to look at how the different landing page cohorts were using the site. The customers to the new landing page were bouncing less and taking different paths through the site, but was that because we had limited the navigation to one principal call-to-action (CTA) where as the original landing page had numerous CTAs. We also theorized that because there was a video on the original landing page that our customers were getting all the information they needed from this video and for a number of people this was a more compelling reason to purchase than our new simplified and more emotionally driven landing page.
With contradictory data we decided to let the test run and sure enough in time the sales began to slowly switch in favor of the new landing page.
Finally we could declare a winner and we could switch to a new landing page.
Our initial data supported classic marketing theory – more engagement = more sales. We discovered on further analysis that the original landing page was reporting more sales because many of the sales reported to “new visitors” were in fact from returning users. We were able to determine this by the fact that many of those “new users” came from internal emails and that the problem was with the cookies or lack of.
Time and further analysis of the data proved that the initial success with engagement was translating in to better sales.
We can declare a winner but make no assumptions
What if the sales numbers had not turned in favor of the new landing page? The new page did change behavior and you can never make assumptions from the initial data. For tests with markedly different creative it is better to have a few data points and if in doubt don’t jump to assumptions and keep digging.
“Queen of the Net” Gives Us “The State of the Net 2011″
Former Morgan Stanley chief Internet analyst, Mary Meeker, presented her annual Internet Report yesterday.
Meeker, who Barron’s famously bestowed the “Queen of the Net” title upon, is always worth listening to as she has great track record of spotting the big Internet trends going back to the mid 90′s.
The big trends documented in the report are:
- The Internet is going global
- The Internet is going mobile
- New User Interfaces are changing how we interact and connect with technology. Think touch screen. Next big thing – audio.
- e-Commerce has a ways to go.
The trajectory is for growth, growth, growth… - Ad $$ will continue to flow towards the media where people are spending more and more of their time – the Internet.
Most of the “big” trends she discusses in her report are well documented but she provides the analysis to support the thesis that these are trends that will impact us all and are not merely short term phenomena.
Some of my favorite nuggets.
- In 3 Years, China Added More Internet Users than Exist in USA.
- Smartphones Surpassed Feature Phone Shipments in W. Europe (Q2:10) / USA (Q1:11)
- For all those who are down on Apple today, look at the adoption of iPad vs iPhone vs iPod.
iPad Growth Leaves “Siblings” in Dust
- The following slide illustrates a point that I have been making for years – there is still a disconnect between where people are spending their time and where the ad $$ are going.
Time Spent with Various Media vs % of Ad $$ Spent on that Media
For the full report -
Going Mobile
The Who sang it in 1971 and in 2011 it is where the Internet is rapidly going.
To put the rise of the mobile Web in to perspective, I pulled the following graph from Coremetrics 4th Annual Online Retail Holiday Readiness Report.
During the 6 month period, Oct. 2010 to Apr. 2011, mobile traffic to e-commerce sites grew from 4.8 % of all traffic to 7.6% of all traffic. That is a 58% increase. Coremetrics estimate that mobile traffic could reach double digits by the end of the year.
Time to Get Mobile
With the ever increasing adoption of Web enabled mobile devices the Web will be untethered from the “traditional” computer browser environment.
With 1 in 10 of your customers potentially reaching your Website from a mobile device by Dec, it is critical that your site is mobile enabled and that your not putting obstacles in the way of some of your most profitable customers.
A Good Starting Point: Econsultancy Mobile Ecosysteme Infographic
Econsultancy put a great infographic together that “explores the trends, companies, organizations and statistics at the center of mobile marketing”.
They also have links to some additional resources from this page.
Inspiration – The Who
And for a little inspiration to help you develop your mobile strategy, The Who, Going Mobile. What it is all about, the freedom of being mobile.
Facebook Serving 31% of all Banner Ads…Really?
According to comScore Ad Metrix’s latest report on display ads for the 1st Q of 2011, Facebook is serving about one out of every three banner ad impressions (31.2%) in the U.S.
No other publisher comes close to Facebook according to comscore. Yahoo is the closest with 10.1%. Google sites only serve 2.5% of the total.
On these numbers Facebook deserves its valuation.
But then I stopped. I don’t remember seeing too many banner ads on Facebook. Could comscore be including the test/image CPC ads that are served on the right hand side of every page.
I requested clarification from comscore and sure enough these text/image ads are included in the calculation of facebook’s banner ad serving capability.
“We collect ad impression data on the ads that appear on the right-hand side of Facebook (text/image ads), as well as any ads served by 3rd party ad servers through Facebook apps (such as in the ones served by the Flixster app below).”
This report indicates that Facebook is the leading network for online display ads and should be included in every display buy.
I don’t think this is the case. How can you compare the banners on the homepage of Yahoo for brand and awareness to a text image ad?
By including these text/image ads as display advertising comscore is inflating the size and value of facebook. What about the test/image ads that are served by other publishers e.g AOL Sponsored Listings. Google also provides this type of inventory on Youtube. This inventory is not included in Comscore’s report.
Facebook is critically important to every online marketing campaign but for display buys, lets not mix text/image ads with banner ads.
The “big data” thesis is that the next “big” opportunity to emerge from the Internet revolution will be the ability to mine and extract profitable insights from the mountain of data that is now being collected digitally.
This is the optimistic projection in a “big” report just released by McKinsey, “Big Data: The Next Frontier for Innovation, Competition and Productivity.” An example of the potential value according to McKinsey is the $300 billion to the US Health Care System.
All well and good but as the report rightly points out, there will be a huge need for people who can mine and extract this value.
From my experience it is not the access to the data or the ability to run complex statistical computations that will be the bottle neck to extracting the value, but the ability to interpret and analyse the data.
1. We need to start with asking the right questions. Are they going to improve the quality of our decisions? Why are we engaging in this exercise? What do we hope to get out of it?
2. When we get back answers from the mass of data, we need to put them in to a broader context. From my experience working with online analytics the data rarely throws up a clear answer. It needs to be interpreted, added to other data, and put in to the context of the business situation.
3. Most importantly and something that will not be found in any statistical package is that we must be willing to follow where the analysis leads us. Too often we get married to an idea or have already predetermined the outcome in our minds. This is on individual level and more difficult to tackle on an institutional level.
Who wants to bring the bad news that our “big idea” is not working. There are a multitude of examples where the data told corporations and institutions that something was wrong but they kept going anyway.
Having access to more data, statistical packages and PHD’s to run the database queries is all useless if we don’t work on honest analysis and are willing to kill “bad” ideas and run with the good ones. Mining the data is going to be easier than changing the human condition.
Is each AOLer worth only 34 cents?
Silicon Alley Insider noted today that AOL’s ad revenues dropped a “startling” 20% in the 1st Q.
Yes 20% is a steep drop but what I found even more startling was this little factoid derived from the numbers:
AOL’s ad revenues in 1st Q = $109 million.
AOL’s average monthly domestic unique visitors = 106 million
This works out that each unique visitor to the AOL network of properties is worth, on average, just 34.3 cents per month in ad revenue.
It’s not as if these visitors are not spending any time with AOL properties. AOL served up “58 billion domestic page views, according to comScore Media Metrix, which translates into 181 average monthly domestic page views per unique visitor.”
181 page views for 34 cents. Assuming there was at least 1 ad on every page this equates to .0018 cents per ad. Some thing is wrong with this equation.
It again points out to me that the Internet industry is not doing a very good job at selling the value of the online consumer. I am sure that the average AOL visitor would put a greater value upon their time and attention than 34 cents. How come advertisers and those selling access to these visitors are not putting a greater value upon them?
This is an insane situation when you compare what advertisers spend to get in front of there customers using other mediums. It is a situation that can not last because like gravitational pull the ad $$ will eventually begin to flow to the medium that more and more people are spending more and more time with.
Don’t worry AOL, better days lie ahead.
That OMG Moment – What Every Marketer Should Strive For
This is a Skype exchange with my wife after she has just realized that shoes that she had ordered from Zappos at 10 pm arrived the very next day.
I was surprised at how quickly they were able to deliver the sneakers even with the complimentary expediated shipping.
Zappos goes beyond providing a service. It works on delighting its customers. These OMG! moments are happening all around the US for Zappos customers. It is why they are getting such great press.
Delivering these types of customer experiences are what great marketing and great companies are made of. Zappos is now expanding beyond shoes in to bags, accessories and electronics. This is a company to watch.
Tactics Are No Substitute for an Internet Strategy
Two online advertising “enhancements” announced this week garnered a lot of attention.
One was the announcement on March 10 that the Online Publishers Assn. (OPA) is going to enable its members to run larger display-ads. The objective of these bigger ads, and they will be a lot bigger, taking up the complete screen for many, is that they will enable brand advertisers to create more engaging ads that leave an impact. The size of the ads will ensure that they will have an impact. Whether they provide a brand lift or merely annoy is another question. These bigger ad units will be coming to a premium content site near you this summer. (Business Week, PaidContent).
The next day, March 11, Google announched it was adding behavioual targeting to its box of algorithmic tricks on how it serves and targets ads. It is instructive that they are launching the new capability on YouTube with a select number of advertiser before rolling out to the wider advertising community. YouTube, despite its huge popularity does not contribute in any measurable way to Google’s profitability. In fact it may be a drag on Google’s profit margins. Newsweek recently did an analysis on YouTube’s revenues and lack of profitability.
In these recessionary times even Google is conscious that business units that are a drag on profits cannot go on indefinitely. Google have maintained for quite some time that monetizing Youtube is a top priority.
At the heart of these two announcements, whether it is new media as represented by Google or old media media as represented by the New York Times and Business Week, is that online publishers are desperately trying to figure out new ways to bring more ad dollars online. For the OPA there is a sense of desperation and for Google, a sense of reality.
What is missing in the respective organizations PR and the backlash from the various consumer watchdog and privacy advocates is that these “enhancements” are only going to gain traction if they are proven effective for advertisers. Neither initiative is going to change the online landscape in any material way. These are merely additional tactics that can be used or not used by advertisers.
Over the years we have added contextual advertising, rich media advertising, re-targeting capabilities and numerous other tactics. Some have gained traction and are widely used by online media planners. Some such as the ill-fated ‘Beacon” initiative from Faceback burnt out very quickly in an onslaught of negative feedback.
What none of these initiatives have provided is a panacea for the single most pressing problem facing the Internet industry - not enough advertisers are willing to pay any form of premium to get there message in front of there audience online. The Internet is just not seen as a medium that can compete with TV or with print for that matter when it comes to brand advertising. A few years ago rich media was hailed as the solution for brand advertisers and there were numerous studies and press releases about how these highly engaging ads were going to divert more of the brand managers budget online. Rich media, while providing for better more appealing ads, did not, despite the hype, change the dynamics of online advertising. Similar to the other enhancements and various add-ons over the years, bigger ads or the addition of behavioral advertising by Google are not going to having any significant impact upon how much of a brand manager’s budget is dedicated to the Internet.
These are all just tactics. They are good to have and they provide the online media planner with a plethora of tools that she can use to develop a comprehensive media plan. These media planning tactics do not substitute for a digital strategy. Until we start developing more campaigns that have a digital strategy as there foundation we are not going to experience any significant shift in brand dollars to the Internet.
A digital strategy is not confined to whether you are going to utilize behavioral ad targeting or rich media or bigger ad units. A strategy encompasses all of the elements that go in to engaging and communicating with your customers in a digital environment. A strategy encompasses your Website, your ads, your media plan and all of its components, your email and your ongoing customer communications, social networking and the two-way conversations that you engage in and lastly the reporting and the analytics that go in to informing and guiding your decisions. This is how you build a brand online.
Takes a lot of bricks to build a Brand online
Digital strategy like brand building takes a lot more work than choosing the tactical elements that go in to your media plan. Until we have more of a focus on strategy and better proof points on how a brand is built online, the Internet is not going to garner a greater percentage of brand dollars .
Building Your Brand in a Recession
This is a great presentation from The Economist on why a brand needs to continue marketing during a downturn if it wants to maintain market share and grow.
It is a bit self-serving, concluding that a rush towards online marketing because of the need for accountability is not going to help build a brand. Online, according to the economist, maintains brand awareness. Print (and as a corollary other offline media) build brands.
Though it is probably true right now, I do not think that this has to remain the case. Digital media can build brands. The problem, as I see it, is that not enough effort and $$ have been applied on how to use the Internet to help build a brand. Intuitively I have always thought that this interactive medium should be able to build a brand by immersing and engaging the consumer in the brand’s story.
The presentation draws on BMW to provide an example of how a brand can grow market share by maintaining spend in a recession if its competitors were to cut spending. BMW was the one brand that attempted to leverage the full potential of the Internet with BMWFilms – a series of 8 highly professional films produced for the Internet. This was back in 2002 before YouTube and broadband were ubiquitious. I can’t recall any brand since creating such a buzz online.
We are long over due for some innovative brand building online.
Warning: you have to view the presentation in full screen mode to read the mircotext.
