Optimarketer: The Marketing Optimization Blog

Marketing success results from trial and error. Not a single try.

Imagine a CEO telling a highly accomplished Sales VP, "I’m giving you one shot at your sales strategy. If what you implement doesn’t work out of the gate, you’re out."

Of course, that scenario rarely happens in rational businesses. But in marketing, here’s an all-too-common scene: An important experiment gets okayed, but those implementing it are given a single test cell – one shot at success. Even when a minimum of say, three, are justified. Limitations of this nature often result in inadequate sample sizes; experiments aimed at inferior targets; and more failed tests.

Marketers who’ve run lots of multi-cell tests have seen a full range of outcomes from cell to cell. What isn’t profitable in one test cell may perform very differently in another.

So when testing, do it right the first time, so you’re not left asking, "What if?" 

What say you?

 

Direct Marketing Legend Succeeded by Failing ... and Failing

You're looking at one of the most profitable direct response ads -- ever. It was conceived and written by Joe Sugarman, living legend of our Facebook group.

Over the years, Joe sold around 20 million pair of BluBlocker sunglasses. These days he lives a Cribs-like existence in Maui, which my brother Allan tells me is Hawaii's most beautiful island. 

I spoke with Joe earlier this year, and he mentioned something that should be burned into the memory of online and offline marketers everywhere.

The man who wrote, "I have failed more times than anybody I have ever known," talked about extreme, consistent, deliberate failure. Joe said his ads failed around 90% of the time.

Marketing Research 101 professors say not to run a test unless the potential value of testing far exceeds the cost. Direct Marketing 101 professors add that tests should be run in ways that minimize downside risk while maximizing upside potential.

Intelligent direct marketers launch tests to thousands and roll out winners to millions. They alter or dump losers before they cost big bucks. Profits of winners are far higher than tests that crash and burn. Simple.

Testing made Joe Sugarman richer than any of the "gurus" who made money talking about getting rich in marketing. It's a lesson worth repeating next time you propose a test and someone asks, "Aren't you already supposed to know the answer?"

Have a great testing story to share?

If at First You Don't Whup the Direct Response Champ ...

"Beating the control" is the art and science of defeating a reigning direct marketing champion: measurable advertising that's been victorious in projectable split-run tests, or advertising that's never been tested head-to-head but has run at an acceptable profit.

Some "control" approaches have been on top for several years, and they're as tough to knock out as Floyd Mayweather. If you've been in the direct marketing game for a while, you know it's impossible to win 100% of the time (and if anyone tells you otherwise they're either naive or full of shit). Mothers of Invention has beaten the control creative/offer approach more than 50% of the time in online and offline media, and I haven't heard of many legitimate groups doin' much better.

Much has been written about general guidelines for beating the control, but one aspect -- what to do after defeat -- has rarely, if ever, been discussed.

After a loss, great prizefighters pick themselves up off the canvas and get back in the gym. Sadly, many marketing managers aren't worthy of adjectives like "great" and tend to essentially say, "Throw da bum out" after a single defeat -- even if they themselves have managed a string of consecutive losses over the years. No, they don't exactly live by the Golden Rule.

The path to victory is actually not terribly mysterious. The boxer -- with his or her trainer -- runs the fight film. If they notice the winner stayed inside (remained close to the opponent), the new plan may include well-timed straight rights to keep the champ at bay. The direct marketing equivalent of this may be a change in the selling proposition. Boxers look for signs from the opponent's earlier bouts; we often have primary or secondary research to guide us.

Our marketing shop has been involved in dozens of record-breaking direct marketing campaigns. One reason: We kick ass in rematches. 

What tips do you have on beating the control?

"You're Right -- It'll Be a Blown Test. Don't Tell the Client."

One day, while working at an agency I'd rather not name, an account executive showed me a direct mail test grid for a consumer client. I wasn't on that particular team but liked seeing how everyone's tests were structured.

As I studied it my jaw dropped. I quickly asked, "Has it run?" He said nothing had moved forward. I breathed a sigh of relief and filled him in on the fatal flaw.

They were trying three offers, but instead of running a controlled test, they went with this approach:

Offer A: Cross-Section of Lists 1-3

Offer B: Cross-Section of Lists 4-6

Offer C: Cross-Section of Lists 7-9

See the problem? By running each offer test on a different set of lists, there was absolutely no way of knowing if differences in outcomes were due to offers or lists. You could drive a truck through the hole in their scheme.

The account executive didn't seem to care, so I went to the guy in charge of the account. He smiled when I shared my observations and said I forgot one important consideration. When I asked "What's that?" he claimed different collections of lists pretty much perform identically. I asked where he picked that up, and realized he was making it up out of thin air.

We went back and forth for a while, and eventually he conceded it was possible for different sets of lists to perform differently if all other factors stayed the same. This meant we both knew we were looking at the makings of a blown test -- a waste of the client's money that could cause an inferior offer to run ... and run.

Instead of picking up the phone, admitting the problem, and offering to shoot over a new test grid, he said something I would never forget: "We don't have to tell the client."

For this and other reasons, I resigned weeks later.

What do you think drives smart people to "manage" client relationships in such stupid ways?

The "Mix Everything Up" Direct Marketing Test

Test only one variable at a time.

This maxim has been pounded into the heads of direct marketers for generations. For good reason.

Direct marketers that don't isolate one variable at a time can't possibly know exactly what causes differences seen in A/B tests. Was it the new format? The offer? They simply don't know.

On the other hand, many direct marketers lack the budget, sample size, time, and patience to experiment with one element at a time.

That's where our "Mix Everything Up" test comes in.

Let's say we've used the same self-mailer for some time, and its recent performance has been, to say the least, underwhelming. We've always wondered if an envelope package would improve the bottom line. Plus the creative concept doesn't seem all that hot. And the offer was never much to write home about.

The budget won't allow us to run all necessary versions to determine which of our test options, if any, will bear fruit. So instead, we test the "control" self-mailer format -- including the current creative approach and offer -- against an envelope package featuring a brand new creative approach and offer.

To pull this test off properly, we split one or more lists in half, either on a random or nth basis. But always remember this, mis amigos: If we test Direct Mail Format/Creative/Offer Combination "A" on List #1 and Direct Mail Format/Creative/Offer Combination "B" on List #2, we will indeed have a blown test. No doubt about it.

Some direct marketing veterans will no doubt argue that even if we properly split lists, it's a pointless exercise, because if "B" beats "A" by 26%, there's no way of knowing exactly what caused the significant difference.

It's absolutely true that we won't understand the contribution of each variable, but if we receive enough responses to have a high confidence level, we'll know that "B" is about 26% better than "A" -- and we'll be able to roll out "B" with confidence.

We'll definitely know that "B" works better. We just won't know exactly why.

What do you so-called quants out there think?

Mo' Testing In Marketing

I love it when marketers do A/B tests of Google AdWords ads for the first time and act like they split the atom.

Split-run tests -- where circulation is carved up to test different advertising approaches -- are hardly new. People like Claude Hopkins (1866-1932) were running A/B tests of different ads when Teddy Roosevelt was president. Hopkins told marketers everywhere about his findings back in 1923 when he released Scientific Advertising.

Too bad more marketers didn't grab a copy.

I've been involved in split-run tests of headlines, offers, positioning, colors, short copy vs. long copy, and even ad size. It's amazing how easy it is to see double- or even triple-digit differences by running projectable tests.

Yet the vast majority of marketers -- including direct marketing  people -- speculate rather than work with facts. They retire successful ads because they think they're "tired." They replace what's made big bucks with what's new because they don't "like" the earlier stuff.

Don't imitate the know-it-alls who don't know it all.